ii

Adaptive Computation and Machine Learning
Francis Bach, series editor

A complete list of books published in the Adaptive Computation and Machine Learning
series appears at the back of this book.

Reinforcement Learning:
An Introduction

second edition

Richard S. Sutton and Andrew G. Barto

The MIT Press
Cambridge, Massachusetts
London, England© 2018, 2020 Richard S. Sutton and Andrew G. Barto
All rights reserved. No part of this book may be reproduced in any form by any electronic
or mechanical means (including photocopying, recording, or information storage and retrieval)without permission in writing from the copyright holder. This work is licensed under theCreative Commons Attribution-NonCommercial-NoDerivs 2.0 Generic License. To view a copyof this license, visit http://creativecommons.org/licenses/by-nc-nd/2.0/ or send a letter
to Creative Commons, PO Box 1866, Mountain View, CA 94042, USA.
This book was set in 10/12, CMR by Westchester Publishing Services. Printed and bound in
the United States of America.
Library of Congress Cataloging-in-Publication Data
Names: Sutton, Richard S., author. | Barto, Andrew G., author.

Title: Reinforcement learning : an introduction / Richard S. Sutton and Andrew G. Barto.
Description: Second edition. | Cambridge, MA : The MIT Press, [2018] | Series: Adaptive
computation and machine learning series | Includes bibliographical references and index.
Identifiers: LCCN 2018023826 | ISBN 9780262039246 (hardcover : alk. paper)
Subjects: LCSH: Reinforcement learning.

Classification: LCC Q325.6 .R45 2018 | DDC 006.3/1--dc23 LC record available
at https://lccn.loc.gov/2018023826
10

9

8

7

6

5

4

3

2

1

In memory of A. Harry Klopf

Contents
Preface to the Second Edition

xiii

Preface to the First Edition

xvii

Summary of Notation

xix

1 Introduction
1
1.1 Reinforcement Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.3 Elements of Reinforcement Learning . . . . . . . . . . . . . . . . . . . . . 6
1.4 Limitations and Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.5 An Extended Example: Tic-Tac-Toe . . . . . . . . . . . . . . . . . . . . . 8
1.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
1.7 Early History of Reinforcement Learning . . . . . . . . . . . . . . . . . . . 13

I

Tabular Solution Methods

2 Multi-armed Bandits
2.1 A k -armed Bandit Problem . . . . . . . . . . . . . . . . . . . . . . . . . .
2.2 Action-value Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.3 The 10-armed Testbed . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.4 Incremental Implementation . . . . . . . . . . . . . . . . . . . . . . . . . .
2.5 Tracking a Nonstationary Problem . . . . . . . . . . . . . . . . . . . . . .
2.6 Optimistic Initial Values . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.7 Upper-Confidence-Bound Action Selection . . . . . . . . . . . . . . . . . .
2.8 Gradient Bandit Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . .
2.9 Associative Search (Contextual Bandits) . . . . . . . . . . . . . . . . . . .
2.10 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

23
25
25
27
28
30
32
34
35
37
41
42

viii

Contents

3 Finite Markov Decision Processes
3.1 The Agent–Environment Interface . . . . . . . . . . . . . . . . . . . . . .
3.2 Goals and Rewards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.3 Returns and Episodes . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.4 Unified Notation for Episodic and Continuing Tasks . . . . . . . . . . . .
3.5 Policies and Value Functions . . . . . . . . . . . . . . . . . . . . . . . . .
3.6 Optimal Policies and Optimal Value Functions . . . . . . . . . . . . . . .
3.7 Optimality and Approximation . . . . . . . . . . . . . . . . . . . . . . . .
3.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

47
47
53
54
57
58
62
67
68

4 Dynamic Programming
4.1 Policy Evaluation (Prediction) . . . . . . . . . . . . . . . . . . . . . . . .
4.2 Policy Improvement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.3 Policy Iteration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.4 Value Iteration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.5 Asynchronous Dynamic Programming . . . . . . . . . . . . . . . . . . . .
4.6 Generalized Policy Iteration . . . . . . . . . . . . . . . . . . . . . . . . . .
4.7 Efficiency of Dynamic Programming . . . . . . . . . . . . . . . . . . . . .
4.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

73
74
76
80
82
85
86
87
88

5 Monte Carlo Methods
91
5.1 Monte Carlo Prediction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
5.2 Monte Carlo Estimation of Action Values . . . . . . . . . . . . . . . . . . 96
5.3 Monte Carlo Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
5.4 Monte Carlo Control without Exploring Starts . . . . . . . . . . . . . . . 100
5.5 O↵-policy Prediction via Importance Sampling . . . . . . . . . . . . . . . 103
5.6 Incremental Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . 109
5.7 O↵-policy Monte Carlo Control . . . . . . . . . . . . . . . . . . . . . . . . 110
5.8 *Discounting-aware Importance Sampling . . . . . . . . . . . . . . . . . . 112
5.9 *Per-decision Importance Sampling . . . . . . . . . . . . . . . . . . . . . . 114
5.10 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
6 Temporal-Di↵erence Learning
119
6.1 TD Prediction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
6.2 Advantages of TD Prediction Methods . . . . . . . . . . . . . . . . . . . . 124
6.3 Optimality of TD(0) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
6.4 Sarsa: On-policy TD Control . . . . . . . . . . . . . . . . . . . . . . . . . 129
6.5 Q-learning: O↵-policy TD Control . . . . . . . . . . . . . . . . . . . . . . 131
6.6 Expected Sarsa . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
6.7 Maximization Bias and Double Learning . . . . . . . . . . . . . . . . . . . 134
6.8 Games, Afterstates, and Other Special Cases . . . . . . . . . . . . . . . . 136
6.9 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138

Contentsix

7 n-step Bootstrapping
141
7.1 n-step TD Prediction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142
7.2 n-step Sarsa . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
7.3 n-step O↵-policy Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . 148
7.4 *Per-decision Methods with Control Variates . . . . . . . . . . . . . . . . 150
7.5 O↵-policy Learning Without Importance Sampling:
The n-step Tree Backup Algorithm . . . . . . . . . . . . . . . . . . . . . . 152
7.6 *A Unifying Algorithm: n-step Q( ) . . . . . . . . . . . . . . . . . . . . . 154
7.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
8 Planning and Learning with Tabular Methods
159
8.1 Models and Planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
8.2 Dyna: Integrated Planning, Acting, and Learning . . . . . . . . . . . . . . 161
8.3 When the Model Is Wrong . . . . . . . . . . . . . . . . . . . . . . . . . . . 166
8.4 Prioritized Sweeping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168
8.5 Expected vs. Sample Updates . . . . . . . . . . . . . . . . . . . . . . . . . 172
8.6 Trajectory Sampling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174
8.7 Real-time Dynamic Programming . . . . . . . . . . . . . . . . . . . . . . . 177
8.8 Planning at Decision Time . . . . . . . . . . . . . . . . . . . . . . . . . . . 180
8.9 Heuristic Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181
8.10 Rollout Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183
8.11 Monte Carlo Tree Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185
8.12 Summary of the Chapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188
8.13 Summary of Part I: Dimensions . . . . . . . . . . . . . . . . . . . . . . . . 189

II

Approximate Solution Methods

195

9 On-policy Prediction with Approximation
197
9.1 Value-function Approximation . . . . . . . . . . . . . . . . . . . . . . . . . 198
9.2 The Prediction Objective (VE) . . . . . . . . . . . . . . . . . . . . . . . . 199
9.3 Stochastic-gradient and Semi-gradient Methods . . . . . . . . . . . . . . . 200
9.4 Linear Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204
9.5 Feature Construction for Linear Methods . . . . . . . . . . . . . . . . . . 210
9.5.1 Polynomials . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210
9.5.2 Fourier Basis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211
9.5.3 Coarse Coding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215
9.5.4 Tile Coding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217
9.5.5 Radial Basis Functions . . . . . . . . . . . . . . . . . . . . . . . . . 221
9.6 Selecting Step-Size Parameters Manually . . . . . . . . . . . . . . . . . . . 222
9.7 Nonlinear Function Approximation: Artificial Neural Networks . . . . . . 223
9.8 Least-Squares TD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 228

xContents

9.9 Memory-based Function Approximation . . . . . . . . . . . . . . . . . . . 230
9.10 Kernel-based Function Approximation . . . . . . . . . . . . . . . . . . . . 232
9.11 Looking Deeper at On-policy Learning: Interest and Emphasis . . . . . . 234
9.12 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236
10 On-policy Control with Approximation
243
10.1 Episodic Semi-gradient Control . . . . . . . . . . . . . . . . . . . . . . . . 243
10.2 Semi-gradient n-step Sarsa . . . . . . . . . . . . . . . . . . . . . . . . . . 247
10.3 Average Reward: A New Problem Setting for Continuing Tasks . . . . . . 249
10.4 Deprecating the Discounted Setting . . . . . . . . . . . . . . . . . . . . . . 253
10.5 Di↵erential Semi-gradient n-step Sarsa . . . . . . . . . . . . . . . . . . . . 255
10.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256
11 *O↵-policy Methods with Approximation257
11.1 Semi-gradient Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258
11.2 Examples of O↵-policy Divergence . . . . . . . . . . . . . . . . . . . . . . 260
11.3 The Deadly Triad . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 264
11.4 Linear Value-function Geometry . . . . . . . . . . . . . . . . . . . . . . . 266
11.5 Gradient Descent in the Bellman Error . . . . . . . . . . . . . . . . . . . . 269
11.6 The Bellman Error is Not Learnable . . . . . . . . . . . . . . . . . . . . . 274
11.7 Gradient-TD Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 278
11.8 Emphatic-TD Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 281
11.9 Reducing Variance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283
11.10 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 284
12 Eligibility Traces287
12.1 The -return . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 288
12.2 TD( ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 292
12.3 n-step Truncated -return Methods . . . . . . . . . . . . . . . . . . . . . 295
12.4 Redoing Updates: Online -return Algorithm . . . . . . . . . . . . . . . . 297
12.5 True Online TD( ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299
12.6 *Dutch Traces in Monte Carlo Learning . . . . . . . . . . . . . . . . . . . 301
12.7 Sarsa( ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303
12.8 Variable and. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 307
12.9 O↵-policy Traces with Control Variates . . . . . . . . . . . . . . . . . . . 309
12.10 Watkins’s Q( ) to Tree-Backup( ) . . . . . . . . . . . . . . . . . . . . . . 312
12.11 Stable O↵-policy Methods with Traces . . . . . . . . . . . . . . . . . . . 314
12.12 Implementation Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 316
12.13 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 317

Contentsxi

13 Policy Gradient Methods
321
13.1 Policy Approximation and its Advantages . . . . . . . . . . . . . . . . . . 322
13.2 The Policy Gradient Theorem . . . . . . . . . . . . . . . . . . . . . . . . . 324
13.3 REINFORCE: Monte Carlo Policy Gradient . . . . . . . . . . . . . . . . . 326
13.4 REINFORCE with Baseline . . . . . . . . . . . . . . . . . . . . . . . . . . 329
13.5 Actor–Critic Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 331
13.6 Policy Gradient for Continuing Problems . . . . . . . . . . . . . . . . . . 333
13.7 Policy Parameterization for Continuous Actions . . . . . . . . . . . . . . . 335
13.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 337

III

Looking Deeper339

14 Psychology
341
14.1 Prediction and Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 342
14.2 Classical Conditioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 343
14.2.1 Blocking and Higher-order Conditioning . . . . . . . . . . . . . . . 345
14.2.2 The Rescorla–Wagner Model . . . . . . . . . . . . . . . . . . . . . 346
14.2.3 The TD Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 349
14.2.4 TD Model Simulations . . . . . . . . . . . . . . . . . . . . . . . . . 350
14.3 Instrumental Conditioning . . . . . . . . . . . . . . . . . . . . . . . . . . . 357
14.4 Delayed Reinforcement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 361
14.5 Cognitive Maps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 363
14.6 Habitual and Goal-directed Behavior . . . . . . . . . . . . . . . . . . . . . 364
14.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 368
15 Neuroscience377
15.1 Neuroscience Basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 378
15.2 Reward Signals, Reinforcement Signals, Values, and Prediction Errors . . 380
15.3 The Reward Prediction Error Hypothesis . . . . . . . . . . . . . . . . . . 381
15.4 Dopamine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 383
15.5 Experimental Support for the Reward Prediction Error Hypothesis . . . . 387
15.6 TD Error/Dopamine Correspondence . . . . . . . . . . . . . . . . . . . . . 390
15.7 Neural Actor–Critic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 395
15.8 Actor and Critic Learning Rules . . . . . . . . . . . . . . . . . . . . . . . 398
15.9 Hedonistic Neurons . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 402
15.10 Collective Reinforcement Learning . . . . . . . . . . . . . . . . . . . . . . 404
15.11 Model-based Methods in the Brain . . . . . . . . . . . . . . . . . . . . . . 407
15.12 Addiction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 409
15.13 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 410

xiiContents

16 Applications and Case Studies
421
16.1 TD-Gammon . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 421
16.2 Samuel’s Checkers Player . . . . . . . . . . . . . . . . . . . . . . . . . . . 426
16.3 Watson’s Daily-Double Wagering . . . . . . . . . . . . . . . . . . . . . . . 429
16.4 Optimizing Memory Control . . . . . . . . . . . . . . . . . . . . . . . . . . 432
16.5 Human-level Video Game Play . . . . . . . . . . . . . . . . . . . . . . . . 436
16.6 Mastering the Game of Go . . . . . . . . . . . . . . . . . . . . . . . . . . . 441
16.6.1 AlphaGo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 444
16.6.2 AlphaGo Zero . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 447
16.7 Personalized Web Services . . . . . . . . . . . . . . . . . . . . . . . . . . . 450
16.8 Thermal Soaring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 453
17 Frontiers459
17.1 General Value Functions and Auxiliary Tasks . . . . . . . . . . . . . . . . 459
17.2 Temporal Abstraction via Options . . . . . . . . . . . . . . . . . . . . . . 461
17.3 Observations and State . . . . . . . . . . . . . . . . . . . . . . . . . . . . 464
17.4 Designing Reward Signals . . . . . . . . . . . . . . . . . . . . . . . . . . . 469
17.5 Remaining Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 472
17.6 Reinforcement Learning and the Future of Artificial Intelligence . . . . . . 475
References

481

Index

519

Preface to the Second Edition
The twenty years since the publication of the first edition of this book have seen tremendous
progress in artificial intelligence, propelled in large part by advances in machine learning,
including advances in reinforcement learning. Although the impressive computational
power that became available is responsible for some of these advances, new developmentsin theory and algorithms have been driving forces as well. In the face of this progress, asecond edition of our 1998 book was long overdue, and we finally began the project in2012. Our goal for the second edition was the same as our goal for the first: to provide aclear and simple account of the key ideas and algorithms of reinforcement learning thatis accessible to readers in all the related disciplines. The edition remains an introduction,
and we retain a focus on core, online learning algorithms. This edition includes some new
topics that rose to importance over the intervening years, and we expanded coverage of
topics that we now understand better. But we made no attempt to provide comprehensive
coverage of the field, which has exploded in many di↵erent directions. We apologize forhaving to leave out all but a handful of these contributions.
As in the first edition, we chose not to produce a rigorous formal treatment ofreinforcement learning, or to formulate it in the most general terms. However, our deeper
understanding of some topics since the first edition required a bit more mathematics
to explain; we have set o↵ the more mathematical parts in shaded boxes that the nonmathematically-inclined may choose to skip. We also use a slightly di↵erent notation
than was used in the first edition. In teaching, we have found that the new notation
helps to address some common points of confusion. It emphasizes the di↵erence between
random variables, denoted with capital letters, and their instantiations, denoted in lower
case. For example, the state, action, and reward at time step t are denoted St , At ,and Rt , while their possible values might be denoted s, a, and r. Along with this, it isnatural to use lower case for value functions (e.g., v⇡ ) and restrict capitals to their tabular
estimates (e.g., Qt (s, a)). Approximate value functions are deterministic functions ofrandom parameters and are thus also in lower case (e.g., v̂(s,wt ) ⇡ v⇡ (s)). Vectors, suchas the weight vector wt (formerly ✓t ) and the feature vector xt (formerly t ), are bold
and written in lowercase even if they are random variables. Uppercase bold is reserved for
matrices. In the first edition we used special notations, Pass0 and Rass0 , for the transition
probabilities and expected rewards. One weakness of that notation is that it still did not
fully characterize the dynamics of the rewards, giving only their expectations, which is
sufficient for dynamic programming but not for reinforcement learning. Another weakness

xiv

Preface to the Second Editionis the excess of subscripts and superscripts. In this edition we use the explicit notation ofp(s0 , r |s, a) for the joint probability for the next state and reward given the current state
and action. All the changes in notation are summarized in a table on page xix.
The second edition is significantly expanded, and its top-level organization has beenchanged. After the introductory first chapter, the second edition is divided into three newparts. The first part (Chapters 2–8) treats as much of reinforcement learning as possible
without going beyond the tabular case for which exact solutions can be found. We coverboth learning and planning methods for the tabular case, as well as their unificationin n-step methods and in Dyna. Many algorithms presented in this part are new tothe second edition, including UCB, Expected Sarsa, Double learning, tree-backup, Q( ),RTDP, and MCTS. Doing the tabular case first, and thoroughly, enables core ideas to be
developed in the simplest possible setting. The second part of the book (Chapters 9–13)
is then devoted to extending the ideas to function approximation. It has new sections on
artificial neural networks, the fourier basis, LSTD, kernel-based methods, Gradient-TD
and Emphatic-TD methods, average-reward methods, true online TD( ), and policygradient methods. The second edition significantly expands the treatment of o↵-policy
learning, first for the tabular case in Chapters 5–7, then with function approximation in
Chapters 11 and 12. Another change is that the second edition separates the forward-viewidea of n-step bootstrapping (now treated more fully in Chapter 7) from the backwardview idea of eligibility traces (now treated independently in Chapter 12). The third partof the book has large new chapters on reinforcement learning’s relationships to psychology(Chapter 14) and neuroscience (Chapter 15), as well as an updated case-studies chapter
including Atari game playing, Watson’s wagering strategy, and the Go playing programs
AlphaGo and AlphaGo Zero (Chapter 16). Still, out of necessity we have included only asmall subset of all that has been done in the field. Our choices reflect our long-standing
interests in inexpensive model-free methods that should scale well to large applications.
The final chapter now includes a discussion of the future societal impacts of reinforcement
learning. For better or worse, the second edition is about twice as large as the first.
This book is designed to be used as the primary text for a one- or two-semester
course on reinforcement learning. For a one-semester course, the first ten chapters should
be covered in order and form a good core, to which can be added material from theother chapters, from other books such as Bertsekas and Tsitsiklis (1996), Wiering andvan Otterlo (2012), and Szepesvári (2010), or from the literature, according to taste.
Depending of the students’ background, some additional material on online supervised
learning may be helpful. The ideas of options and option models are a natural addition(Sutton, Precup and Singh, 1999). A two-semester course can cover all the chapters aswell as supplementary material. The book can also be used as part of broader courses
on machine learning, artificial intelligence, or neural networks. In this case, it may bedesirable to cover only a subset of the material. We recommend covering Chapter 1 for abrief overview, Chapter 2 through Section 2.4, Chapter 3, and then selecting sections
from the remaining chapters according to time and interests. Chapter 6 is the mostimportant for the subject and for the rest of the book. A course focusing on machinelearning or neural networks should cover Chapters 9 and 10, and a course focusing on
artificial intelligence or planning should cover Chapter 8. Throughout the book, sections
and chapters that are more difficult and not essential to the rest of the book are marked

Preface to the Second Edition

xv

with a ⇤. These can be omitted on first reading without creating problems later on. Some
exercises are also marked with a ⇤ to indicate that they are more advanced and notessential to understanding the basic material of the chapter.
Most chapters end with a section entitled “Bibliographical and Historical Remarks,”
wherein we credit the sources of the ideas presented in that chapter, provide pointers tofurther reading and ongoing research, and describe relevant historical background. Despite
our attempts to make these sections authoritative and complete, we have undoubtedly left
out some important prior work. For that we again apologize, and we welcome corrections
and extensions for incorporation into the electronic version of the book.
Like the first edition, this edition of the book is dedicated to the memory of A. Harry
Klopf. It was Harry who introduced us to each other, and it was his ideas about the brain
and artificial intelligence that launched our long excursion into reinforcement learning.
Trained in neurophysiology and long interested in machine intelligence, Harry was asenior scientist affiliated with the Avionics Directorate of the Air Force Office of Scientific
Research (AFOSR) at Wright-Patterson Air Force Base, Ohio. He was dissatisfied withthe great importance attributed to equilibrium-seeking processes, including homeostasis
and error-correcting pattern classification methods, in explaining natural intelligenceand in providing a basis for machine intelligence. He noted that systems that try to
maximize something (whatever that might be) are qualitatively di↵erent from equilibriumseeking systems, and he argued that maximizing systems hold the key to understanding
important aspects of natural intelligence and for building artificial intelligences. Harry was
instrumental in obtaining funding from AFOSR for a project to assess the scientific meritof these and related ideas. This project was conducted in the late 1970s at the Universityof Massachusetts Amherst (UMass Amherst), initially under the direction of Michael
Arbib, William Kilmer, and Nico Spinelli, professors in the Department of Computerand Information Science at UMass Amherst, and founding members of the Cybernetics
Center for Systems Neuroscience at the University, a farsighted group focusing on theintersection of neuroscience and artificial intelligence. Barto, a recent PhD from the
University of Michigan, was hired as post doctoral researcher on the project. Meanwhile,
Sutton, an undergraduate studying computer science and psychology at Stanford, hadbeen corresponding with Harry regarding their mutual interest in the role of stimulus
timing in classical conditioning. Harry suggested to the UMass group that Sutton would
be a great addition to the project. Thus, Sutton became a UMass graduate student,
whose PhD was directed by Barto, who had become an Associate Professor. The studyof reinforcement learning as presented in this book is rightfully an outcome of thatproject instigated by Harry and inspired by his ideas. Further, Harry was responsible
for bringing us, the authors, together in what has been a long and enjoyable interaction.
By dedicating this book to Harry we honor his essential contributions, not only to thefield of reinforcement learning, but also to our collaboration. We also thank Professors
Arbib, Kilmer, and Spinelli for the opportunity they provided to us to begin exploringthese ideas. Finally, we thank AFOSR for generous support over the early years of ourresearch, and the NSF for its generous support over many of the following years.
We have very many people to thank for their inspiration and help with this second
edition. Everyone we acknowledged for their inspiration and help with the first edition

xvi

Preface to the Second Edition

deserve our deepest gratitude for this edition as well, which would not exist were it notfor their contributions to edition number one. To that long list we must add many others
who contributed specifically to the second edition. Our students over the many years that
we have taught this material contributed in countless ways: exposing errors, o↵ering fixes,
and—not the least—being confused in places where we could have explained things better.
We especially thank Martha Steenstrup for reading and providing detailed comments
throughout. The chapters on psychology and neuroscience could not have been written
without the help of many experts in those fields. We thank John Moore for his patient
tutoring over many many years on animal learning experiments, theory, and neuroscience,and for his careful reading of multiple drafts of Chapters 14 and 15. We also thank Matt
Botvinick, Nathaniel Daw, Peter Dayan, and Yael Niv for their penetrating comments ondrafts of these chapters, their essential guidance through the massive literature, and theirinterception of many of our errors in early drafts. Of course, the remaining errors in thesechapters—and there must still be some—are totally our own. We thank Phil Thomas forhelping us make these chapters accessible to non-psychologists and non-neuroscientists,
and we thank Peter Sterling for helping us improve the exposition. We are grateful to Jim
Houk for introducing us to the subject of information processing in the basal ganglia andfor alerting us to other relevant aspects of neuroscience. José Martı́nez, Terry Sejnowski,
David Silver, Gerry Tesauro, Georgios Theocharous, and Phil Thomas generously helped
us understand details of their reinforcement learning applications for inclusion in thecase-studies chapter, and they provided helpful comments on drafts of these sections.
Special thanks are owed to David Silver for helping us better understand Monte Carlo
Tree Search and the DeepMind Go-playing programs. We thank George Konidaris for his
help with the section on the Fourier basis. Emilio Cartoni, Thomas Cederborg, Stefan
Dernbach, Clemens Rosenbaum, Patrick Taylor, Thomas Colin, and Pierre-Luc Baconhelped us in a number important ways for which we are most grateful.
Sutton would also like to thank the members of the Reinforcement Learning andArtificial Intelligence laboratory at the University of Alberta for contributions to thesecond edition. He owes a particular debt to Rupam Mahmood for essential contributions
to the treatment of o↵-policy Monte Carlo methods in Chapter 5, to Hamid Maei for
helping develop the perspective on o↵-policy learning presented in Chapter 11, to Eric
Graves for conducting the experiments in Chapter 13, to Shangtong Zhang for replicating
and thus verifying almost all the experimental results, to Kris De Asis for improving
the new technical content of Chapters 7 and 12, and to Harm van Seijen for insightsthat led to the separation of n-step methods from eligibility traces and (along with Hado
van Hasselt) for the ideas involving exact equivalence of forward and backward views ofeligibility traces presented in Chapter 12. Sutton also gratefully acknowledges the support
and freedom he was granted by the Government of Alberta and the National Science andEngineering Research Council of Canada throughout the period during which the second
edition was conceived and written. In particular, he would like to thank Randy Goebel
for creating a supportive and far-sighted environment for research in Alberta. He would
also like to thank DeepMind their support in the last six months of writing the book.
Finally, we owe thanks to the many careful readers of drafts of the second edition that
we posted on the internet. They found many errors that we had missed and alerted us topotential points of confusion.

Preface to the First Edition
We first came to focus on what is now known as reinforcement learning in late 1979. We
were both at the University of Massachusetts, working on one of the earliest projects torevive the idea that networks of neuronlike adaptive elements might prove to be a promising
approach to artificial adaptive intelligence. The project explored the “heterostatic theoryof adaptive systems” developed by A. Harry Klopf. Harry’s work was a rich source ofideas, and we were permitted to explore them critically and compare them with the long
history of prior work in adaptive systems. Our task became one of teasing the ideas apartand understanding their relationships and relative importance. This continues today,
but in 1979 we came to realize that perhaps the simplest of the ideas, which had long
been taken for granted, had received surprisingly little attention from a computational
perspective. This was simply the idea of a learning system that wants something, that
adapts its behavior in order to maximize a special signal from its environment. This
was the idea of a “hedonistic” learning system, or, as we would say now, the idea ofreinforcement learning.
Like others, we had a sense that reinforcement learning had been thoroughly exploredin the early days of cybernetics and artificial intelligence. On closer inspection, though,we found that it had been explored only slightly. While reinforcement learning had clearlymotivated some of the earliest computational studies of learning, most of these researchers
had gone on to other things, such as pattern classification, supervised learning, andadaptive control, or they had abandoned the study of learning altogether. As a result, thespecial issues involved in learning how to get something from the environment received
relatively little attention. In retrospect, focusing on this idea was the critical step that
set this branch of research in motion. Little progress could be made in the computational
study of reinforcement learning until it was recognized that such a fundamental idea had
not yet been thoroughly explored.
The field has come a long way since then, evolving and maturing in several directions.
Reinforcement learning has gradually become one of the most active research areas in machine learning, artificial intelligence, and neural network research. The field has developed
strong mathematical foundations and impressive applications. The computational studyof reinforcement learning is now a large field, with hundreds of active researchers aroundthe world in diverse disciplines such as psychology, control theory, artificial intelligence,and neuroscience. Particularly important have been the contributions establishing anddeveloping the relationships to the theory of optimal control and dynamic programming.

xviii

Preface to the First Edition

The overall problem of learning from interaction to achieve goals is still far from being
solved, but our understanding of it has improved significantly. We can now place component ideas, such as temporal-di↵erence learning, dynamic programming, and function
approximation, within a coherent perspective with respect to the overall problem.
Our goal in writing this book was to provide a clear and simple account of the key
ideas and algorithms of reinforcement learning. We wanted our treatment to be accessible
to readers in all of the related disciplines, but we could not cover all of these perspectives
in detail. For the most part, our treatment takes the point of view of artificial intelligenceand engineering. Coverage of connections to other fields we leave to others or to anothertime. We also chose not to produce a rigorous formal treatment of reinforcement learning.
We did not reach for the highest possible level of mathematical abstraction and did not
rely on a theorem–proof format. We tried to choose a level of mathematical detail thatpoints the mathematically inclined in the right directions without distracting from thesimplicity and potential generality of the underlying ideas.
In some sense we have been working toward this book for thirty years, and we have lotsof people to thank. First, we thank those who have personally helped us develop the overall
view presented in this book: Harry Klopf, for helping us recognize that reinforcement
learning needed to be revived; Chris Watkins, Dimitri Bertsekas, John Tsitsiklis, andPaul Werbos, for helping us see the value of the relationships to dynamic programming;
John Moore and Jim Kehoe, for insights and inspirations from animal learning theory;
Oliver Selfridge, for emphasizing the breadth and importance of adaptation; and, moregenerally, our colleagues and students who have contributed in countless ways: Ron
Williams, Charles Anderson, Satinder Singh, Sridhar Mahadevan, Steve Bradtke, Bob
Crites, Peter Dayan, and Leemon Baird. Our view of reinforcement learning has beensignificantly enriched by discussions with Paul Cohen, Paul Utgo↵, Martha Steenstrup,
Gerry Tesauro, Mike Jordan, Leslie Kaelbling, Andrew Moore, Chris Atkeson, Tom
Mitchell, Nils Nilsson, Stuart Russell, Tom Dietterich, Tom Dean, and Bob Narendra.
We thank Michael Littman, Gerry Tesauro, Bob Crites, Satinder Singh, and Wei Zhang
for providing specifics of Sections 4.7, 15.1, 15.4, 15.5, and 15.6 respectively. We thankthe Air Force Office of Scientific Research, the National Science Foundation, and GTE
Laboratories for their long and farsighted support.
We also wish to thank the many people who have read drafts of this book andprovided valuable comments, including Tom Kalt, John Tsitsiklis, Pawel Cichosz, OlleGällmo, Chuck Anderson, Stuart Russell, Ben Van Roy, Paul Steenstrup, Paul Cohen,
Sridhar Mahadevan, Jette Randlov, Brian Sheppard, Thomas O’Connell, Richard Coggins,
Cristina Versino, John H. Hiett, Andreas Badelt, Jay Ponte, Joe Beck, Justus Piater,
Martha Steenstrup, Satinder Singh, Tommi Jaakkola, Dimitri Bertsekas, Torbjörn Ekman,
Christina Björkman, Jakob Carlström, and Olle Palmgren. Finally, we thank Gwyn
Mitchell for helping in many ways, and Harry Stanton and Bob Prior for being ourchampions at MIT Press.

Summary of Notation
Capital letters are used for random variables, whereas lower case letters are used forthe values of random variables and for scalar functions. Quantities that are required tobe real-valued vectors are written in bold and in lower case (even if random variables).
Matrices are bold capitals.
.
=
equality relationship that is true by definition⇡
approximately equal/
proportional to
Pr{X = x}probability that a random variable X takes on the value x.
X⇠prandom variable X selected from distribution p(x)P= Pr{X = x}.
E[X]expectation of a random variable X, i.e., E[X] = x p(x)x
argmaxa f (a) a value of a at which f (a) takes its maximal valueln x
natural logarithm of x
ex , exp(x)the base of the natural logarithm, e ⇡ 2.71828, carried to power x; eln x = x
R
set of real numbers
f :X!Yfunction f from elements of set X to elements of set Y
assignment(a, b]
the real interval between a and b including b but not including a"↵,

predicateprobability of taking a random action in an "-greedy policy
step-size parameters
discount-rate parameter
decay-rate parameter for eligibility traces.
indicator function ( predicate = 1 if the predicate is true, else 0)

In a multi-arm bandit problem:knumber of actions (arms)tdiscrete time step or play numberq⇤ (a)
true value (expected reward) of action a
Qt (a)
estimate at time t of q⇤ (a)Nt (a)
number of times action a has been selected up prior to time t
Ht (a)
learned preference for selecting action a at time t⇡t (a)
probability of selecting action a at time tR̄testimate at time t of the expected reward given ⇡t

xxSummary of Notation

In a Markov Decision Process:s, s0
statesaan actionra rewardS
set of all nonterminal statesS+
set of all states, including the terminal stateA(s)set of all actions available in state sR
set of all possible rewards, a finite subset of R⇢
subset of (e.g., R ⇢ R)
2
is an element of; e.g. (s 2 S, r 2 R)
|S|
number of elements in set StT, T (t)
At
St
Rt
⇡
⇡(s)
⇡(a|s)

discrete time step
final time step of an episode, or of the episode including time step t
action at time tstate at time t, typically due, stochastically, to St 1 and At 1
reward at time t, typically due, stochastically, to St 1 and At 1
policy (decision-making rule)action taken in state s under deterministic policy ⇡probability of taking action a in state s under stochastic policy ⇡Gth
Gt:t+n , Gt:h
Ḡt:h
Gt
Gt:h
Gt s , Gt a

return following time thorizon, the time step one looks up to in a forward viewn-step return from t + 1 to t + n, or to h (discounted and corrected)
flat return (undiscounted and uncorrected) from t + 1 to h (Section 5.8)
-return (Section 12.1)
truncated, corrected -return (Section 12.3)-return, corrected by estimated state, or action, values (Section 12.8)p(s0 , r |s, a)p(s0 |s, a)r(s, a)r(s, a, s0 )probability of transition to state s0 with reward r, from state s and action aprobability of transition to state s0 , from state s taking action a
expected immediate reward from state s after action a
expected immediate reward on transition from s to s0 under action av⇡ (s)v⇤ (s)q⇡ (s, a)q⇤ (s, a)value of state s under policy ⇡ (expected return)value of state s under the optimal policy
value of taking action a in state s under policy ⇡value of taking action a in state s under the optimal policy

V, Vt
Q, Qt
V̄t (s)Utarray estimates of state-value function v⇡ or v⇤
array estimates of action-value function q⇡ or q⇤. P
expected approximate action value; for example, V̄t (s) = a ⇡(a|s)Qt (s, a)target for estimate at time t

Summary of Notation

t
s a
t, t

n
d
d0
w, wt
wi , wt,iv̂(s,w)vw (s)q̂(s, a, w)
rv̂(s,w)rq̂(s, a, w)

xxi

temporal-di↵erence (TD) error at t (a random variable) (Section 6.1)
state- and action-specific forms of the TD error (Section 12.9)
in n-step methods, n is the number of steps of bootstrapping
dimensionality—the number of components of w
alternate dimensionality—the number of components of ✓d-vector of weights underlying an approximate value function
ith component of learnable weight vector
approximate value of state s given weight vector w
alternate notation for v̂(s,w)
approximate value of state–action pair s, a given weight vector w
column vector of partial derivatives of v̂(s,w) with respect to w
column vector of partial derivatives of q̂(s, a, w) with respect to w

x(s)vector of features visible when in state sx(s, a)vector of features visible when in state s taking action axi (s), xi (s, a) ith component of vector x(s) or x(s, a)
xt
shorthand for x(St ) or x(St , At ) P.
.
w> x
inner product of vectors, w> x = i wi xi ; for example, v̂(s,w) = w> x(s)v, vt
secondary d-vector of weights, used to learn w (Chapter 11)
zt
d-vector of eligibility traces at time t (Chapter 12)✓, ✓t⇡(a|s, ✓)⇡✓r⇡(a|s, ✓)
J(✓)
rJ(✓)
h(s, a, ✓)parameter vector of target policy (Chapter 13)
probability of taking action a in state s given parameter vector ✓policy corresponding to parameter ✓column vector of partial derivatives of ⇡(a|s, ✓) with respect to ✓performance measure for the policy ⇡✓column vector of partial derivatives of J(✓) with respect to ✓preference for selecting action a in state s based on ✓b(a|s)
b(s)
b
⇢t:h
⇢t
r(⇡)
R̄t

behavior policy used to select actions while learning about target policy ⇡a baseline function b : S 7! R for policy-gradient methods
branching factor for an MDP or search tree
importance sampling ratio for time t through time h (Section 5.5).
importance sampling ratio for time t alone, ⇢t = ⇢t:t
average reward (reward rate) for policy ⇡ (Section 10.3)
estimate of r(⇡) at time t

µ(s)
µ
2
kvkµ
⌘(s)
⇧
B⇡

on-policy distribution over states (Section 9.2)
|S|-vector of the µ(s) for all s 2 S
2 . Pµ-weighted squared norm of value function v, i.e., kvkµ = s2S µ(s)v(s)2
expected number of visits to state s per episode (page 199)
projection operator for value functions (page 268)
Bellman operator for value functions (Section 11.4)

xxii

A
b
wTD
I
P
D
X
¯w (s)¯w , BE
VE(w)
BE(w)
PBE(w)
TDE(w)
RE(w)Summary of Notationhi>.
d ⇥ d matrix A = E xt xt
xt+1.
d-dimensional vector b = E[Rt+1 xt ].
TD fixed point wTD = A 1 b (a d-vector, Section 9.4)
identity matrix|S| ⇥ |S| matrix of state-transition probabilities under ⇡|S| ⇥ |S| diagonal matrix with µ on its diagonal|S| ⇥ d matrix with the x(s) as its rows
Bellman error (expected TD error) for vw at state s (Section 11.4)
Bellman error vector, with components ¯w (s).
2mean square value error VE(w) = kvw v⇡ kµ (Section 9.2)
2.
mean square Bellman error BE(w) = ¯w µ2.
mean square projected Bellman error PBE(w) = ⇧ ¯w µ⇥⇤.
mean square temporal-di↵erence error TDE(w) = Eb ⇢t t2 (Section 11.5)
mean square return error (Section 11.6)

Chapter 1

Introduction
The idea that we learn by interacting with our environment is probably the first to occur
to us when we think about the nature of learning. When an infant plays, waves its arms,
or looks about, it has no explicit teacher, but it does have a direct sensorimotor connection
to its environment. Exercising this connection produces a wealth of information about
cause and e↵ect, about the consequences of actions, and about what to do in order toachieve goals. Throughout our lives, such interactions are undoubtedly a major sourceof knowledge about our environment and ourselves. Whether we are learning to drivea car or to hold a conversation, we are acutely aware of how our environment responds
to what we do, and we seek to influence what happens through our behavior. Learningfrom interaction is a foundational idea underlying nearly all theories of learning andintelligence.
In this book we explore a computational approach to learning from interaction. Rather
than directly theorizing about how people or animals learn, we primarily explore idealized
learning situations and evaluate the e↵ectiveness of various learning methods. Thatis, we adopt the perspective of an artificial intelligence researcher or engineer. We
explore designs for machines that are e↵ective in solving learning problems of scientific or
economic interest, evaluating the designs through mathematical analysis or computational
experiments. The approach we explore, called reinforcement learning, is much more
focused on goal-directed learning from interaction than are other approaches to machinelearning.

1.1

Reinforcement Learning

Reinforcement learning is learning what to do—how to map situations to actions—so
as to maximize a numerical reward signal. The learner is not told which actions totake, but instead must discover which actions yield the most reward by trying them. In
the most interesting and challenging cases, actions may a↵ect not only the immediate
reward but also the next situation and, through that, all subsequent rewards. These two
characteristics—trial-and-error search and delayed reward—are the two most importantdistinguishing features of reinforcement learning.

2

Chapter 1: Introduction

Reinforcement learning, like many topics whose names end with “ing,” such as machinelearning and mountaineering, is simultaneously a problem, a class of solution methods
that work well on the problem, and the field that studies this problem and its solution
methods. It is convenient to use a single name for all three things, but at the same timeessential to keep the three conceptually separate. In particular, the distinction between
problems and solution methods is very important in reinforcement learning; failing to
make this distinction is the source of many confusions.
We formalize the problem of reinforcement learning using ideas from dynamical systems theory, specifically, as the optimal control of incompletely-known Markov decision
processes. The details of this formalization must wait until Chapter 3, but the basic idea
is simply to capture the most important aspects of the real problem facing a learning
agent interacting over time with its environment to achieve a goal. A learning agent
must be able to sense the state of its environment to some extent and must be able totake actions that a↵ect the state. The agent also must have a goal or goals relating to
the state of the environment. Markov decision processes are intended to include justthese three aspects—sensation, action, and goal—in their simplest possible forms without
trivializing any of them. Any method that is well suited to solving such problems weconsider to be a reinforcement learning method.
Reinforcement learning is di↵erent from supervised learning, the kind of learning studiedin most current research in the field of machine learning. Supervised learning is learningfrom a training set of labeled examples provided by a knowledgable external supervisor.
Each example is a description of a situation together with a specification—the label—of
the correct action the system should take in that situation, which is often to identify acategory to which the situation belongs. The object of this kind of learning is for thesystem to extrapolate, or generalize, its responses so that it acts correctly in situations
not present in the training set. This is an important kind of learning, but alone it is notadequate for learning from interaction. In interactive problems it is often impractical toobtain examples of desired behavior that are both correct and representative of all thesituations in which the agent has to act. In uncharted territory—where one would expect
learning to be most beneficial—an agent must be able to learn from its own experience.
Reinforcement learning is also di↵erent from what machine learning researchers call
unsupervised learning, which is typically about finding structure hidden in collections of
unlabeled data. The terms supervised learning and unsupervised learning would seemto exhaustively classify machine learning paradigms, but they do not. Although one
might be tempted to think of reinforcement learning as a kind of unsupervised learning
because it does not rely on examples of correct behavior, reinforcement learning is trying
to maximize a reward signal instead of trying to find hidden structure. Uncovering
structure in an agent’s experience can certainly be useful in reinforcement learning, but byitself does not address the reinforcement learning problem of maximizing a reward signal.
We therefore consider reinforcement learning to be a third machine learning paradigm,
alongside supervised learning and unsupervised learning and perhaps other paradigms.

1.1. Reinforcement Learning3

One of the challenges that arise in reinforcement learning, and not in other kindsof learning, is the trade-o↵ between exploration and exploitation. To obtain a lot of
reward, a reinforcement learning agent must prefer actions that it has tried in the pastand found to be e↵ective in producing reward. But to discover such actions, it has totry actions that it has not selected before. The agent has to exploit what it has alreadyexperienced in order to obtain reward, but it also has to explore in order to make better
action selections in the future. The dilemma is that neither exploration nor exploitation
can be pursued exclusively without failing at the task. The agent must try a variety ofactions and progressively favor those that appear to be best. On a stochastic task, eachaction must be tried many times to gain a reliable estimate of its expected reward. The
exploration–exploitation dilemma has been intensively studied by mathematicians formany decades, yet remains unresolved. For now, we simply note that the entire issue ofbalancing exploration and exploitation does not even arise in supervised and unsupervised
learning, at least in the purest forms of these paradigms.
Another key feature of reinforcement learning is that it explicitly considers the wholeproblem of a goal-directed agent interacting with an uncertain environment. This is incontrast to many approaches that consider subproblems without addressing how theymight fit into a larger picture. For example, we have mentioned that many machinelearning researchers have studied supervised learning without specifying how such anability would ultimately be useful. Other researchers have developed theories of planningwith general goals, but without considering planning’s role in real-time decision making,or the question of where the predictive models necessary for planning would come from.
Although these approaches have yielded many useful results, their focus on isolated
subproblems is a significant limitation.
Reinforcement learning takes the opposite tack, starting with a complete, interactive,
goal-seeking agent. All reinforcement learning agents have explicit goals, can sense
aspects of their environments, and can choose actions to influence their environments.
Moreover, it is usually assumed from the beginning that the agent has to operate despite
significant uncertainty about the environment it faces. When reinforcement learning
involves planning, it has to address the interplay between planning and real-time action
selection, as well as the question of how environment models are acquired and improved.
When reinforcement learning involves supervised learning, it does so for specific reasonsthat determine which capabilities are critical and which are not. For learning research to
make progress, important subproblems have to be isolated and studied, but they should
be subproblems that play clear roles in complete, interactive, goal-seeking agents, even if
all the details of the complete agent cannot yet be filled in.
By a complete, interactive, goal-seeking agent we do not always mean something like
a complete organism or robot. These are clearly examples, but a complete, interactive,
goal-seeking agent can also be a component of a larger behaving system. In this case, theagent directly interacts with the rest of the larger system and indirectly interacts withthe larger system’s environment. A simple example is an agent that monitors the charge
level of robot’s battery and sends commands to the robot’s control architecture. This
agent’s environment is the rest of the robot together with the robot’s environment. It is

4

Chapter 1: Introduction

important to look beyond the most obvious examples of agents and their environments
to appreciate the generality of the reinforcement learning framework.
One of the most exciting aspects of modern reinforcement learning is its substantive
and fruitful interactions with other engineering and scientific disciplines. Reinforcement
learning is part of a decades-long trend within artificial intelligence and machine learning
toward greater integration with statistics, optimization, and other mathematical subjects.
For example, the ability of some reinforcement learning methods to learn with parameterized approximators addresses the classical “curse of dimensionality” in operations researchand control theory. More distinctively, reinforcement learning has also interacted strongly
with psychology and neuroscience, with substantial benefits going both ways. Of all theforms of machine learning, reinforcement learning is the closest to the kind of learningthat humans and other animals do, and many of the core algorithms of reinforcement
learning were originally inspired by biological learning systems. Reinforcement learning
has also given back, both through a psychological model of animal learning that better
matches some of the empirical data, and through an influential model of parts of thebrain’s reward system. The body of this book develops the ideas of reinforcement learningthat pertain to engineering and artificial intelligence, with connections to psychology andneuroscience summarized in Chapters 14 and 15.
Finally, reinforcement learning is also part of a larger trend in artificial intelligenceback toward simple general principles. Since the late 1960s, many artificial intelligence researchers presumed that there are no general principles to be discovered, that intelligence isinstead due to the possession of a vast number of special purpose tricks, procedures, andheuristics. It was sometimes said that if we could just get enough relevant facts into amachine, say one million, or one billion, then it would become intelligent. Methods basedon general principles, such as search or learning, were characterized as “weak methods,”
whereas those based on specific knowledge were called “strong methods.” This view isuncommon today. From our point of view, it was premature: too little e↵ort had been
put into the search for general principles to conclude that there were none. Modern
artificial intelligence now includes much research looking for general principles of learning,search, and decision making. It is not clear how far back the pendulum will swing, but
reinforcement learning research is certainly part of the swing back toward simpler andfewer general principles of artificial intelligence.

1.2

ExamplesA good way to understand reinforcement learning is to consider some of the examples
and possible applications that have guided its development.
• A master chess player makes a move. The choice is informed both by planning—
anticipating possible replies and counterreplies—and by immediate, intuitive judgments of the desirability of particular positions and moves.
• An adaptive controller adjusts parameters of a petroleum refinery’s operation inreal time. The controller optimizes the yield/cost/quality trade-o↵ on the basis

1.2. Examples

5of specified marginal costs without sticking strictly to the set points originallysuggested by engineers.
• A gazelle calf struggles to its feet minutes after being born. Half an hour later it isrunning at 20 miles per hour.
• A mobile robot decides whether it should enter a new room in search of more trash
to collect or start trying to find its way back to its battery recharging station. It
makes its decision based on the current charge level of its battery and how quickly
and easily it has been able to find the recharger in the past.
• Phil prepares his breakfast. Closely examined, even this apparently mundane
activity reveals a complex web of conditional behavior and interlocking goal–subgoal
relationships: walking to the cupboard, opening it, selecting a cereal box, thenreaching for, grasping, and retrieving the box. Other complex, tuned, interactive
sequences of behavior are required to obtain a bowl, spoon, and milk carton. Eachstep involves a series of eye movements to obtain information and to guide reaching
and locomotion. Rapid judgments are continually made about how to carry theobjects or whether it is better to ferry some of them to the dining table before
obtaining others. Each step is guided by goals, such as grasping a spoon or getting
to the refrigerator, and is in service of other goals, such as having the spoon to eat
with once the cereal is prepared and ultimately obtaining nourishment. Whether
he is aware of it or not, Phil is accessing information about the state of his bodythat determines his nutritional needs, level of hunger, and food preferences.
These examples share features that are so basic that they are easy to overlook. All
involve interaction between an active decision-making agent and its environment, within
which the agent seeks to achieve a goal despite uncertainty about its environment. The
agent’s actions are permitted to a↵ect the future state of the environment (e.g., thenext chess position, the level of reservoirs of the refinery, the robot’s next location andthe future charge level of its battery), thereby a↵ecting the actions and opportunities
available to the agent at later times. Correct choice requires taking into account indirect,
delayed consequences of actions, and thus may require foresight or planning.
At the same time, in all of these examples the e↵ects of actions cannot be fully predicted;
thus the agent must monitor its environment frequently and react appropriately. For
example, Phil must watch the milk he pours into his cereal bowl to keep it from overflowing.
All these examples involve goals that are explicit in the sense that the agent can judge
progress toward its goal based on what it can sense directly. The chess player knows
whether or not he wins, the refinery controller knows how much petroleum is being
produced, the gazelle calf knows when it falls, the mobile robot knows when its batteries
run down, and Phil knows whether or not he is enjoying his breakfast.
In all of these examples the agent can use its experience to improve its performanceover time. The chess player refines the intuition he uses to evaluate positions, thereby
improving his play; the gazelle calf improves the efficiency with which it can run; Phillearns to streamline making his breakfast. The knowledge the agent brings to the task at
the start—either from previous experience with related tasks or built into it by design or6Chapter 1: Introduction

evolution—influences what is useful or easy to learn, but interaction with the environmentis essential for adjusting behavior to exploit specific features of the task.

1.3

Elements of Reinforcement Learning

Beyond the agent and the environment, one can identify four main subelements of areinforcement learning system: a policy, a reward signal , a value function, and, optionally,a model of the environment.
A policy defines the learning agent’s way of behaving at a given time. Roughly speaking,a policy is a mapping from perceived states of the environment to actions to be taken
when in those states. It corresponds to what in psychology would be called a set ofstimulus–response rules or associations. In some cases the policy may be a simple function
or lookup table, whereas in others it may involve extensive computation such as a search
process. The policy is the core of a reinforcement learning agent in the sense that it alone
is sufficient to determine behavior. In general, policies may be stochastic, specifying
probabilities for each action.
A reward signal defines the goal of a reinforcement learning problem. On each time
step, the environment sends to the reinforcement learning agent a single number called
the reward. The agent’s sole objective is to maximize the total reward it receives overthe long run. The reward signal thus defines what are the good and bad events for theagent. In a biological system, we might think of rewards as analogous to the experiencesof pleasure or pain. They are the immediate and defining features of the problem facedby the agent. The reward signal is the primary basis for altering the policy; if an action
selected by the policy is followed by low reward, then the policy may be changed to
select some other action in that situation in the future. In general, reward signals may
be stochastic functions of the state of the environment and the actions taken.
Whereas the reward signal indicates what is good in an immediate sense, a valuefunction specifies what is good in the long run. Roughly speaking, the value of a state is
the total amount of reward an agent can expect to accumulate over the future, startingfrom that state. Whereas rewards determine the immediate, intrinsic desirability ofenvironmental states, values indicate the long-term desirability of states after taking into
account the states that are likely to follow and the rewards available in those states. For
example, a state might always yield a low immediate reward but still have a high value
because it is regularly followed by other states that yield high rewards. Or the reverse
could be true. To make a human analogy, rewards are somewhat like pleasure (if high)and pain (if low), whereas values correspond to a more refined and farsighted judgmentof how pleased or displeased we are that our environment is in a particular state.
Rewards are in a sense primary, whereas values, as predictions of rewards, are secondary.
Without rewards there could be no values, and the only purpose of estimating values is toachieve more reward. Nevertheless, it is values with which we are most concerned when
making and evaluating decisions. Action choices are made based on value judgments. We
seek actions that bring about states of highest value, not highest reward, because theseactions obtain the greatest amount of reward for us over the long run. Unfortunately, it
is much harder to determine values than it is to determine rewards. Rewards are basicallygiven directly by the environment, but values must be estimated and re-estimated from

1.4. Limitations and Scope7

the sequences of observations an agent makes over its entire lifetime. In fact, the mostimportant component of almost all reinforcement learning algorithms we consider is amethod for efficiently estimating values. The central role of value estimation is arguably
the most important thing that has been learned about reinforcement learning over thelast six decades.
The fourth and final element of some reinforcement learning systems is a model ofthe environment. This is something that mimics the behavior of the environment, ormore generally, that allows inferences to be made about how the environment will behave.
For example, given a state and action, the model might predict the resultant next stateand next reward. Models are used for planning, by which we mean any way of deciding
on a course of action by considering possible future situations before they are actuallyexperienced. Methods for solving reinforcement learning problems that use models andplanning are called model-based methods, as opposed to simpler model-free methods thatare explicitly trial-and-error learners—viewed as almost the opposite of planning. In
Chapter 8 we explore reinforcement learning systems that simultaneously learn by trial
and error, learn a model of the environment, and use the model for planning. Modern
reinforcement learning spans the spectrum from low-level, trial-and-error learning tohigh-level, deliberative planning.

1.4

Limitations and Scope

Reinforcement learning relies heavily on the concept of state—as input to the policy andvalue function, and as both input to and output from the model. Informally, we canthink of the state as a signal conveying to the agent some sense of “how the environmentis” at a particular time. The formal definition of state as we use it here is given by
the framework of Markov decision processes presented in Chapter 3. More generally,
however, we encourage the reader to follow the informal meaning and think of the state
as whatever information is available to the agent about its environment. In e↵ect, we
assume that the state signal is produced by some preprocessing system that is nominallypart of the agent’s environment. We do not address the issues of constructing, changing,or learning the state signal in this book (other than briefly in Section 17.3). We take thisapproach not because we consider state representation to be unimportant, but in orderto focus fully on the decision-making issues. In other words, our concern in this book is
not with designing the state signal, but with deciding what action to take as a functionof whatever state signal is available.
Most of the reinforcement learning methods we consider in this book are structured
around estimating value functions, but it is not strictly necessary to do this to solve
reinforcement learning problems. For example, solution methods such as genetic algorithms, genetic programming, simulated annealing, and other optimization methods neverestimate value functions. These methods apply multiple static policies each interacting
over an extended period of time with a separate instance of the environment. The policies
that obtain the most reward, and random variations of them, are carried over to thenext generation of policies, and the process repeats. We call these evolutionary methods
because their operation is analogous to the way biological evolution produces organisms

8

Chapter 1: Introduction

with skilled behavior even if they do not learn during their individual lifetimes. If thespace of policies is sufficiently small, or can be structured so that good policies arecommon or easy to find—or if a lot of time is available for the search—then evolutionary
methods can be e↵ective. In addition, evolutionary methods have advantages on problems
in which the learning agent cannot sense the complete state of its environment.
Our focus is on reinforcement learning methods that learn while interacting with theenvironment, which evolutionary methods do not do. Methods able to take advantageof the details of individual behavioral interactions can be much more efficient thanevolutionary methods in many cases. Evolutionary methods ignore much of the usefulstructure of the reinforcement learning problem: they do not use the fact that the policy
they are searching for is a function from states to actions; they do not notice which states
an individual passes through during its lifetime, or which actions it selects. In some casessuch information can be misleading (e.g., when states are misperceived), but more often it
should enable more efficient search. Although evolution and learning share many featuresand naturally work together, we do not consider evolutionary methods by themselves tobe especially well suited to reinforcement learning problems and, accordingly, we do notcover them in this book.

1.5

An Extended Example: Tic-Tac-Toe

To illustrate the general idea of reinforcement learning and contrast it with other approaches, we next consider a single example in more detail.
Consider the familiar child’s game of tic-tac-toe. Two players
take turns playing on a three-by-three board. One player plays
Xs and the other Os until one player wins by placing three marks
in a row, horizontally, vertically, or diagonally, as the X player
has in the game shown to the right. If the board fills up withneither player getting three in a row, then the game is a draw.
Because a skilled player can play so as never to lose, let us assume
that we are playing against an imperfect player, one whose play
is sometimes incorrect and allows us to win. For the moment, infact, let us consider draws and losses to be equally bad for us. How might we construct aplayer that will find the imperfections in its opponent’s play and learn to maximize its
chances of winning?
Although this is a simple problem, it cannot readily be solved in a satisfactory way
through classical techniques. For example, the classical “minimax” solution from gametheory is not correct here because it assumes a particular way of playing by the opponent.
For example, a minimax player would never reach a game state from which it could
lose, even if in fact it always won from that state because of incorrect play by the
opponent. Classical optimization methods for sequential decision problems, such as
dynamic programming, can compute an optimal solution for any opponent, but require
as input a complete specification of that opponent, including the probabilities with which
the opponent makes each move in each board state. Let us assume that this information
is not available a priori for this problem, as it is not for the vast majority of problems of

X O O
O X X
X

1.5. An Extended Example: Tic-Tac-Toe

9practical interest. On the other hand, such information can be estimated from experience,in this case by playing many games against the opponent. About the best one can doon this problem is first to learn a model of the opponent’s behavior, up to some level ofconfidence, and then apply dynamic programming to compute an optimal solution given
the approximate opponent model. In the end, this is not that di↵erent from some of thereinforcement learning methods we examine later in this book.
An evolutionary method applied to this problem would directly search the spaceof possible policies for one with a high probability of winning against the opponent.
Here, a policy is a rule that tells the player what move to make for every state of thegame—every possible configuration of Xs and Os on the three-by-three board. For eachpolicy considered, an estimate of its winning probability would be obtained by playing
some number of games against the opponent. This evaluation would then direct which
policy or policies were considered next. A typical evolutionary method would hill-climb
in policy space, successively generating and evaluating policies in an attempt to obtain
incremental improvements. Or, perhaps, a genetic-style algorithm could be used thatwould maintain and evaluate a population of policies. Literally hundreds of di↵erent
optimization methods could be applied.
Here is how the tic-tac-toe problem would be approached with a method making useof a value function. First we would set up a table of numbers, one for each possible stateof the game. Each number will be the latest estimate of the probability of our winningfrom that state. We treat this estimate as the state’s value, and the whole table is thelearned value function. State A has higher value than state B, or is considered “better”
than state B, if the current estimate of the probability of our winning from A is higher
than it is from B. Assuming we always play Xs, then for all states with three Xs in a row
the probability of winning is 1, because we have already won. Similarly, for all stateswith three Os in a row, or that are filled up, the correct probability is 0, as we cannot
win from them. We set the initial values of all the other states to 0.5, representing aguess that we have a 50% chance of winning.
We then play many games against the opponent. To select our moves we examine thestates that would result from each of our possible moves (one for each blank space on theboard) and look up their current values in the table. Most of the time we move greedily,
selecting the move that leads to the state with greatest value, that is, with the highest
estimated probability of winning. Occasionally, however, we select randomly from among
the other moves instead. These are called exploratory moves because they cause us toexperience states that we might otherwise never see. A sequence of moves made andconsidered during a game can be diagrammed as in Figure 1.1.
While we are playing, we change the values of the states in which we find ourselves
during the game. We attempt to make them more accurate estimates of the probabilities
of winning. To do this, we “back up” the value of the state after each greedy move tothe state before the move, as suggested by the arrows in Figure 1.1. More precisely, thecurrent value of the earlier state is updated to be closer to the value of the later state.
This can be done by moving the earlier state’s value a fraction of the way toward thevalue of the later state. If we let St denote the state before the greedy move, and St+1
the state after that move, then the update to the estimated value of St , denoted V (St ),10

Chapter 1: Introduction

opponent's move

our move

opponent's move

our move

opponent's move

our move

{
{
{
{
{
{

startingposition
positionstarting
aa

•

•b

b

•

cc c*c*

•d
e*e*

d

•ee
f

•f

•

g gg*
g*
…

..
.

Figure 1.1: A sequence of tic-tac-toe moves. The solid black lines represent the moves taken
during a game; the dashed lines represent moves that we (our reinforcement learning player)
considered but did not make. The * indicates the move currently estimated to be the best. Our
second move was an exploratory move, meaning that it was taken even though another sibling
move, the one leading to e⇤ , was ranked higher. Exploratory moves do not result in any learning,but each of our other moves does, causing updates as suggested by the red arrows in which
estimated values are moved up the tree from later nodes to earlier nodes as detailed in the text.

can be written as
V (St )hV (St ) + ↵ V (St+1 )i
V (St ) ,where ↵ is a small positive fraction called the step-size parameter, which influences
the rate of learning. This update rule is an example of a temporal-di↵erence learning
method, so called because its changes are based on a di↵erence, V (St+1 ) V (St ), between
estimates at two successive times.
The method described above performs quite well on this task. For example, if thestep-size parameter is reduced properly over time, then this method converges, for any
fixed opponent, to the true probabilities of winning from each state given optimal playby our player. Furthermore, the moves then taken (except on exploratory moves) are infact the optimal moves against this (imperfect) opponent. In other words, the method
converges to an optimal policy for playing the game against this opponent. If the step-size
parameter is not reduced all the way to zero over time, then this player also plays wellagainst opponents that slowly change their way of playing.

1.5. An Extended Example: Tic-Tac-Toe

11This example illustrates the di↵erences between evolutionary methods and methods
that learn value functions. To evaluate a policy, an evolutionary method holds the policy
fixed and plays many games against the opponent or simulates many games using a modelof the opponent. The frequency of wins gives an unbiased estimate of the probabilityof winning with that policy, and can be used to direct the next policy selection. But
each policy change is made only after many games, and only the final outcome of eachgame is used: what happens during the games is ignored. For example, if the player wins,
then all of its behavior in the game is given credit, independently of how specific moves
might have been critical to the win. Credit is even given to moves that never occurred!
Value function methods, in contrast, allow individual states to be evaluated. In the end,
evolutionary and value function methods both search the space of policies, but learning avalue function takes advantage of information available during the course of play.
This simple example illustrates some of the key features of reinforcement learning
methods. First, there is the emphasis on learning while interacting with an environment,in this case with an opponent player. Second, there is a clear goal, and correct behavior
requires planning or foresight that takes into account delayed e↵ects of one’s choices. For
example, the simple reinforcement learning player would learn to set up multi-move traps
for a shortsighted opponent. It is a striking feature of the reinforcement learning solutionthat it can achieve the e↵ects of planning and lookahead without using a model of the
opponent and without conducting an explicit search over possible sequences of futurestates and actions.
While this example illustrates some of the key features of reinforcement learning, it isso simple that it might give the impression that reinforcement learning is more limited
than it really is. Although tic-tac-toe is a two-person game, reinforcement learning
also applies in the case in which there is no external adversary, that is, in the case ofa “game against nature.” Reinforcement learning also is not restricted to problems inwhich behavior breaks down into separate episodes, like the separate games of tic-tac-toe,with reward only at the end of each episode. It is just as applicable when behavior
continues indefinitely and when rewards of various magnitudes can be received at any
time. Reinforcement learning is also applicable to problems that do not even break down
into discrete time steps like the plays of tic-tac-toe. The general principles apply tocontinuous-time problems as well, although the theory gets more complicated and weomit it from this introductory treatment.
Tic-tac-toe has a relatively small, finite state set, whereas reinforcement learning canbe used when the state set is very large, or even infinite. For example, Gerry Tesauro(1992, 1995) combined the algorithm described above with an artificial neural network tolearn to play backgammon, which has approximately 1020 states. With this many states
it is impossible ever to experience more than a small fraction of them. Tesauro’s program
learned to play far better than any previous program and eventually better than theworld’s best human players (see Section 16.1). The artificial neural network provides the
program with the ability to generalize from its experience, so that in new states it selects
moves based on information saved from similar states faced in the past, as determinedby the network. How well a reinforcement learning system can work in problems withsuch large state sets is intimately tied to how appropriately it can generalize from past

12

Chapter 1: Introduction

experience. It is in this role that we have the greatest need for supervised learning
methods within reinforcement learning. Artificial neural networks and deep learning(Section 9.7) are not the only, or necessarily the best, way to do this.
In this tic-tac-toe example, learning started with no prior knowledge beyond therules of the game, but reinforcement learning by no means entails a tabula rasa view oflearning and intelligence. On the contrary, prior information can be incorporated into
reinforcement learning in a variety of ways that can be critical for efficient learning (e.g.,
see Sections 9.5, 17.4, and 13.1). We also have access to the true state in the tic-tac-toe
example, whereas reinforcement learning can also be applied when part of the state is
hidden, or when di↵erent states appear to the learner to be the same.
Finally, the tic-tac-toe player was able to look ahead and know the states that would
result from each of its possible moves. To do this, it had to have a model of the gamethat allowed it to foresee how its environment would change in response to moves that it
might never make. Many problems are like this, but in others even a short-term modelof the e↵ects of actions is lacking. Reinforcement learning can be applied in either case.
A model is not required, but models can easily be used if they are available or can belearned (Chapter 8).
On the other hand, there are reinforcement learning methods that do not need any
kind of environment model at all. Model-free systems cannot even think about how
their environments will change in response to a single action. The tic-tac-toe player ismodel-free in this sense with respect to its opponent: it has no model of its opponentof any kind. Because models have to be reasonably accurate to be useful, model-freemethods can have advantages over more complex methods when the real bottleneck insolving a problem is the difficulty of constructing a sufficiently accurate environment
model. Model-free methods are also important building blocks for model-based methods.
In this book we devote several chapters to model-free methods before we discuss how
they can be used as components of more complex model-based methods.
Reinforcement learning can be used at both high and low levels in a system. Although
the tic-tac-toe player learned only about the basic moves of the game, nothing prevents
reinforcement learning from working at higher levels where each of the “actions” mayitself be the application of a possibly elaborate problem-solving method. In hierarchicallearning systems, reinforcement learning can work simultaneously on several levels.
Exercise 1.1: Self-Play Suppose, instead of playing against a random opponent, thereinforcement learning algorithm described above played against itself, with both sides
learning. What do you think would happen in this case? Would it learn a di↵erent policy
for selecting moves?
⇤
Exercise 1.2: Symmetries Many tic-tac-toe positions appear di↵erent but are reallythe same because of symmetries. How might we amend the learning process describedabove to take advantage of this? In what ways would this change improve the learning
process? Now think again. Suppose the opponent did not take advantage of symmetries.
In that case, should we? Is it true, then, that symmetrically equivalent positions shouldnecessarily have the same value?
⇤
Exercise 1.3: Greedy Play Suppose the reinforcement learning player was greedy, that is,
it always played the move that brought it to the position that it rated the best. Might it1.7. Early History of Reinforcement Learning

13

learn to play better, or worse, than a nongreedy player? What problems might occur? ⇤Exercise 1.4: Learning from Exploration Suppose learning updates occurred after all
moves, including exploratory moves. If the step-size parameter is appropriately reduced
over time (but not the tendency to explore), then the state values would converge toa di↵erent set of probabilities. What (conceptually) are the two sets of probabilities
computed when we do, and when we do not, learn from exploratory moves? Assumingthat we do continue to make exploratory moves, which set of probabilities might be betterto learn? Which would result in more wins?
⇤
Exercise 1.5: Other Improvements Can you think of other ways to improve the reinforcement learning player? Can you think of any better way to solve the tic-tac-toe problem
as posed?
⇤

1.6

Summary

Reinforcement learning is a computational approach to understanding and automating
goal-directed learning and decision making. It is distinguished from other computational
approaches by its emphasis on learning by an agent from direct interaction with its
environment, without requiring exemplary supervision or complete models of the environment. In our opinion, reinforcement learning is the first field to seriously address thecomputational issues that arise when learning from interaction with an environment inorder to achieve long-term goals.
Reinforcement learning uses the formal framework of Markov decision processes todefine the interaction between a learning agent and its environment in terms of states,
actions, and rewards. This framework is intended to be a simple way of representing
essential features of the artificial intelligence problem. These features include a sense of
cause and e↵ect, a sense of uncertainty and nondeterminism, and the existence of explicit
goals.
The concepts of value and value function are key to most of the reinforcement learning
methods that we consider in this book. We take the position that value functions
are important for efficient search in the space of policies. The use of value functions
distinguishes reinforcement learning methods from evolutionary methods that search
directly in policy space guided by evaluations of entire policies.

1.7

Early History of Reinforcement Learning

The early history of reinforcement learning has two main threads, both long and rich, thatwere pursued independently before intertwining in modern reinforcement learning. One
thread concerns learning by trial and error, and originated in the psychology of animallearning. This thread runs through some of the earliest work in artificial intelligenceand led to the revival of reinforcement learning in the early 1980s. The second thread
concerns the problem of optimal control and its solution using value functions anddynamic programming. For the most part, this thread did not involve learning. The
two threads were mostly independent, but became interrelated to some extent around a

14

Chapter 1: Introduction

third, less distinct thread concerning temporal-di↵erence methods such as that used in
the tic-tac-toe example in this chapter. All three threads came together in the late 1980s
to produce the modern field of reinforcement learning as we present it in this book.
The thread focusing on trial-and-error learning is the one with which we are mostfamiliar and about which we have the most to say in this brief history. Before doing that,
however, we briefly discuss the optimal control thread.
The term “optimal control” came into use in the late 1950s to describe the problem ofdesigning a controller to minimize or maximize a measure of a dynamical system’s behavior
over time. One of the approaches to this problem was developed in the mid-1950s by
Richard Bellman and others through extending a nineteenth century theory of Hamilton
and Jacobi. This approach uses the concepts of a dynamical system’s state and of avalue function, or “optimal return function,” to define a functional equation, now often
called the Bellman equation. The class of methods for solving optimal control problems
by solving this equation came to be known as dynamic programming (Bellman, 1957a).
Bellman (1957b) also introduced the discrete stochastic version of the optimal control
problem known as Markov decision processes (MDPs). Ronald Howard (1960) devised
the policy iteration method for MDPs. All of these are essential elements underlying thetheory and algorithms of modern reinforcement learning.
Dynamic programming is widely considered the only feasible way of solving general
stochastic optimal control problems. It su↵ers from what Bellman called “the curse ofdimensionality,” meaning that its computational requirements grow exponentially with
the number of state variables, but it is still far more efficient and more widely applicablethan any other general method. Dynamic programming has been extensively developedsince the late 1950s, including extensions to partially observable MDPs (surveyed by
Lovejoy, 1991), many applications (surveyed by White, 1985, 1988, 1993), approximation
methods (surveyed by Rust, 1996), and asynchronous methods (Bertsekas, 1982, 1983).
Many excellent modern treatments of dynamic programming are available (e.g., Bertsekas,
2005, 2012; Puterman, 1994; Ross, 1983; Whittle, 1982, 1983). Bryson (1996) provides
an authoritative history of optimal control.
Connections between optimal control and dynamic programming, on the one hand,and learning, on the other, were slow to be recognized. We cannot be sure about what
accounted for this separation, but its main cause was likely the separation between
the disciplines involved and their di↵erent goals. Also contributing may have been theprevalent view of dynamic programming as an o↵-line computation depending essentiallyon accurate system models and analytic solutions to the Bellman equation. Further,
the simplest form of dynamic programming is a computation that proceeds backwards
in time, making it difficult to see how it could be involved in a learning process thatmust proceed in a forward direction. Some of the earliest work in dynamic programming,such as that by Bellman and Dreyfus (1959), might now be classified as following
a learning approach. Witten’s (1977) work (discussed below) certainly qualifies as a
combination of learning and dynamic-programming ideas. Werbos (1987) argued explicitly
for greater interrelation of dynamic programming and learning methods and for dynamic
programming’s relevance to understanding neural and cognitive mechanisms. For us thefull integration of dynamic programming methods with online learning did not occur

1.7. Early History of Reinforcement Learning

15

until the work of Chris Watkins in 1989, whose treatment of reinforcement learning using
the MDP formalism has been widely adopted. Since then these relationships have been
extensively developed by many researchers, most particularly by Dimitri Bertsekas andJohn Tsitsiklis (1996), who coined the term “neurodynamic programming” to refer tothe combination of dynamic programming and artificial neural networks. Another term
currently in use is “approximate dynamic programming.” These various approaches
emphasize di↵erent aspects of the subject, but they all share with reinforcement learning
an interest in circumventing the classical shortcomings of dynamic programming.
We consider all of the work in optimal control also to be, in a sense, work in reinforcement learning. We define a reinforcement learning method as any e↵ective way of solving
reinforcement learning problems, and it is now clear that these problems are closely
related to optimal control problems, particularly stochastic optimal control problems
such as those formulated as MDPs. Accordingly, we must consider the solution methodsof optimal control, such as dynamic programming, also to be reinforcement learning
methods. Because almost all of the conventional methods require complete knowledgeof the system to be controlled, it feels a little unnatural to say that they are part ofreinforcement learning. On the other hand, many dynamic programming algorithms are
incremental and iterative. Like learning methods, they gradually reach the correct answer
through successive approximations. As we show in the rest of this book, these similarities
are far more than superficial. The theories and solution methods for the cases of complete
and incomplete knowledge are so closely related that we feel they must be considered
together as part of the same subject matter.
Let us return now to the other major thread leading to the modern field of reinforcement
learning, the thread centered on the idea of trial-and-error learning. We only touch on
the major points of contact here, taking up this topic in more detail in Section 14.3.
According to American psychologist R. S. Woodworth (1938) the idea of trial-and-error
learning goes as far back as the 1850s to Alexander Bain’s discussion of learning by“groping and experiment” and more explicitly to the British ethologist and psychologist
Conway Lloyd Morgan’s 1894 use of the term to describe his observations of animal
behavior. Perhaps the first to succinctly express the essence of trial-and-error learning asa principle of learning was Edward Thorndike:
Of several responses made to the same situation, those which are accompaniedor closely followed by satisfaction to the animal will, other things being
equal, be more firmly connected with the situation, so that, when it recurs,
they will be more likely to recur; those which are accompanied or closely
followed by discomfort to the animal will, other things being equal, have theirconnections with that situation weakened, so that, when it recurs, they willbe less likely to occur. The greater the satisfaction or discomfort, the greater
the strengthening or weakening of the bond. (Thorndike, 1911, p. 244)
Thorndike called this the “Law of E↵ect” because it describes the e↵ect of reinforcing
events on the tendency to select actions. Thorndike later modified the law to betteraccount for subsequent data on animal learning (such as di↵erences between the e↵ectsof reward and punishment), and the law in its various forms has generated considerable
controversy among learning theorists (e.g., see Gallistel, 2005; Herrnstein, 1970; Kimble,

16

Chapter 1: Introduction

1961, 1967; Mazur, 1994). Despite this, the Law of E↵ect—in one form or another—iswidely regarded as a basic principle underlying much behavior (e.g., Hilgard and Bower,
1975; Dennett, 1978; Campbell, 1960; Cziko, 1995). It is the basis of the influential
learning theories of Clark Hull (1943, 1952) and the influential experimental methods of
B. F. Skinner (1938).
The term “reinforcement” in the context of animal learning came into use well after
Thorndike’s expression of the Law of E↵ect, first appearing in this context (to the best of
our knowledge) in the 1927 English translation of Pavlov’s monograph on conditioned
reflexes. Pavlov described reinforcement as the strengthening of a pattern of behavior due
to an animal receiving a stimulus—a reinforcer—in an appropriate temporal relationshipwith another stimulus or with a response. Some psychologists extended the idea ofreinforcement to include weakening as well as strengthening of behavior, and extended
the idea of a reinforcer to include possibly the omission or termination of stimulus. To beconsidered a reinforcer, the strengthening or weakening must persist after the reinforcer
is withdrawn; a stimulus that merely attracts an animal’s attention or that energizes its
behavior without producing lasting changes would not be considered a reinforcer.
The idea of implementing trial-and-error learning in a computer appeared among theearliest thoughts about the possibility of artificial intelligence. In a 1948 report, Alan
Turing described a design for a “pleasure-pain system” that worked along the lines of the
Law of E↵ect:
When a configuration is reached for which the action is undetermined, arandom choice for the missing data is made and the appropriate entry is madein the description, tentatively, and is applied. When a pain stimulus occurs
all tentative entries are cancelled, and when a pleasure stimulus occurs they
are all made permanent. (Turing, 1948)
Many ingenious electro-mechanical machines were constructed that demonstrated trialand-error learning. The earliest may have been a machine built by Thomas Ross (1933)
that was able to find its way through a simple maze and remember the path throughthe settings of switches. In 1951 W. Grey Walter built a version of his “mechanicaltortoise” (Walter, 1950) capable of a simple form of learning. In 1952 Claude Shannon
demonstrated a maze-running mouse named Theseus that used trial and error to findits way through a maze, with the maze itself remembering the successful directions
via magnets and relays under its floor (see also Shannon, 1951). J. A. Deutsch (1954)
described a maze-solving machine based on his behavior theory (Deutsch, 1953) thathas some properties in common with model-based reinforcement learning (Chapter 8).
In his PhD dissertation, Marvin Minsky (1954) discussed computational models of
reinforcement learning and described his construction of an analog machine composed ofcomponents he called SNARCs (Stochastic Neural-Analog Reinforcement Calculators)
meant to resemble modifiable synaptic connections in the brain (Chapter 15). The
web site cyberneticzoo.com contains a wealth of information on these and many otherelectro-mechanical learning machines.
Building electro-mechanical learning machines gave way to programming digital computers to perform various types of learning, some of which implemented trial-and-error
learning. Farley and Clark (1954) described a digital simulation of a neural-network1.7. Early History of Reinforcement Learning

17

learning machine that learned by trial and error. But their interests soon shifted fromtrial-and-error learning to generalization and pattern recognition, that is, from reinforcement learning to supervised learning (Clark and Farley, 1955). This began a patternof confusion about the relationship between these types of learning. Many researchers
seemed to believe that they were studying reinforcement learning when they were actually
studying supervised learning. For example, artificial neural network pioneers such as
Rosenblatt (1962) and Widrow and Ho↵ (1960) were clearly motivated by reinforcement
learning—they used the language of rewards and punishments—but the systems theystudied were supervised learning systems suitable for pattern recognition and perceptual
learning. Even today, some researchers and textbooks minimize or blur the distinction
between these types of learning. For example, some textbooks have used the term “trialand-error” to describe artificial neural networks that learn from training examples. Thisis an understandable confusion because these networks use error information to update
connection weights, but this misses the essential character of trial-and-error learning as
selecting actions on the basis of evaluative feedback that does not rely on knowledge ofwhat the correct action should be.
Partly as a result of these confusions, research into genuine trial-and-error learning
became rare in the 1960s and 1970s, although there were notable exceptions. In the 1960s
the terms “reinforcement” and “reinforcement learning” were used in the engineering
literature for the first time to describe engineering uses of trial-and-error learning (e.g.,
Waltz and Fu, 1965; Mendel, 1966; Fu, 1970; Mendel and McClaren, 1970). Particularly
influential was Minsky’s paper “Steps Toward Artificial Intelligence” (Minsky, 1961),
which discussed several issues relevant to trial-and-error learning, including prediction,
expectation, and what he called the basic credit-assignment problem for complex reinforcement learning systems: How do you distribute credit for success among the manydecisions that may have been involved in producing it? All of the methods we discuss inthis book are, in a sense, directed toward solving this problem. Minsky’s paper is well
worth reading today.
In the next few paragraphs we discuss some of the other exceptions and partial
exceptions to the relative neglect of computational and theoretical study of genuine
trial-and-error learning in the 1960s and 1970s.
One exception was the work of the New Zealand researcher John Andreae, whodeveloped a system called STeLLA that learned by trial and error in interaction with
its environment. This system included an internal model of the world and, later, an“internal monologue” to deal with problems of hidden state (Andreae, 1963, 1969; Andreae
and Cashin, 1969). Andreae’s later work (1977) placed more emphasis on learningfrom a teacher, but still included learning by trial and error, with the generation ofnovel events being one of the system’s goals. A feature of this work was a “leakback
process,” elaborated more fully in Andreae (1998), that implemented a credit-assignment
mechanism similar to the backing-up update operations that we describe. Unfortunately,
his pioneering research was not well known and did not greatly impact subsequent
reinforcement learning research. Recent summaries are available (Andreae, 2017a,b).
More influential was the work of Donald Michie. In 1961 and 1963 he described asimple trial-and-error learning system for learning how to play tic-tac-toe (or naughts

18

Chapter 1: Introduction

and crosses) called MENACE (for Matchbox Educable Naughts and Crosses Engine). It
consisted of a matchbox for each possible game position, each matchbox containing a
number of colored beads, a di↵erent color for each possible move from that position. By
drawing a bead at random from the matchbox corresponding to the current game position,
one could determine MENACE’s move. When a game was over, beads were added toor removed from the boxes used during play to reward or punish MENACE’s decisions.
Michie and Chambers (1968) described another tic-tac-toe reinforcement learner called
GLEE (Game Learning Expectimaxing Engine) and a reinforcement learning controller
called BOXES. They applied BOXES to the task of learning to balance a pole hinged toa movable cart on the basis of a failure signal occurring only when the pole fell or the
cart reached the end of a track. This task was adapted from the earlier work of Widrow
and Smith (1964), who used supervised learning methods, assuming instruction from ateacher already able to balance the pole. Michie and Chambers’s version of pole-balancing
is one of the best early examples of a reinforcement learning task under conditions ofincomplete knowledge. It influenced much later work in reinforcement learning, beginningwith some of our own studies (Barto, Sutton, and Anderson, 1983; Sutton, 1984). Michie(1974) consistently emphasized trial and error and learning as essential aspects of artificial
intelligence.
Widrow, Gupta, and Maitra (1973) modified the Least-Mean-Square (LMS) algorithmof Widrow and Ho↵ (1960) to produce a reinforcement learning rule that could learn
from success and failure signals instead of from training examples. They called this formof learning “selective bootstrap adaptation” and described it as “learning with a critic”
instead of “learning with a teacher.” They analyzed this rule and showed how it could
learn to play blackjack. This was an isolated foray into reinforcement learning by Widrow,whose contributions to supervised learning were much more influential. Our use of theterm “critic” is derived from Widrow, Gupta, and Maitra’s paper. Buchanan, Mitchell,
Smith, and Johnson (1978) independently used the term critic in the context of machinelearning (see also Dietterich and Buchanan, 1984), but for them a critic was an expert
system able to do more than evaluate performance.
Research on learning automata had a more direct influence on the trial-and-error
thread leading to modern reinforcement learning research. These are methods for solving
a nonassociative, purely selectional learning problem known as the k-armed bandit byanalogy to a slot machine, or “one-armed bandit,” except with k levers (see Chapter 2).
Learning automata are simple, low-memory machines for improving the probabilityof reward in these problems. Learning automata originated with work in the 1960sof the Russian mathematician and physicist M. L. Tsetlin and colleagues (publishedposthumously in Tsetlin, 1973) and has been extensively developed since then withinengineering (see Narendra and Thathachar, 1974, 1989). These developments included thestudy of stochastic learning automata, which are methods for updating action probabilities
on the basis of reward signals. Although not developed in the tradition of stochastic
learning automata, Harth and Tzanakou’s (1974) Alopex algorithm (for Algorithm ofpattern extraction) is a stochastic method for detecting correlations between actions andreinforcement that influenced some of our early research (Barto, Sutton, and Brouwer,1981). Stochastic learning automata were foreshadowed by earlier work in psychology,
beginning with William Estes’ (1950) e↵ort toward a statistical theory of learning andfurther developed by others (e.g., Bush and Mosteller, 1955; Sternberg, 1963).

1.7. Early History of Reinforcement Learning

19

The statistical learning theories developed in psychology were adopted by researchers ineconomics, leading to a thread of research in that field devoted to reinforcement learning.
This work began in 1973 with the application of Bush and Mosteller’s learning theory to
a collection of classical economic models (Cross, 1973). One goal of this research was tostudy artificial agents that act more like real people than do traditional idealized economic
agents (Arthur, 1991). This approach expanded to the study of reinforcement learningin the context of game theory. Reinforcement learning in economics developed largely
independently of the early work in reinforcement learning in artificial intelligence, thoughgame theory remains a topic of interest in both fields (beyond the scope of this book).
Camerer (2011) discusses the reinforcement learning tradition in economics, and Nowé,Vrancx, and De Hauwere (2012) provide an overview of the subject from the point of viewof multi-agent extensions to the approach that we introduce in this book. Reinforcement
learning in the context of game theory is a much di↵erent subject than reinforcement
learning used in programs to play tic-tac-toe, checkers, and other recreational games. See,for example, Szita (2012) for an overview of this aspect of reinforcement learning andgames.
John Holland (1975) outlined a general theory of adaptive systems based on selectional
principles. His early work concerned trial and error primarily in its nonassociative
form, as in evolutionary methods and the k -armed bandit. In 1976 and more fully in
1986, he introduced classifier systems, true reinforcement learning systems including
association and value functions. A key component of Holland’s classifier systems was
the “bucket-brigade algorithm” for credit assignment, which is closely related to thetemporal di↵erence algorithm used in our tic-tac-toe example and discussed in Chapter 6.
Another key component was a genetic algorithm, an evolutionary method whose role wasto evolve useful representations. Classifier systems have been extensively developed bymany researchers to form a major branch of reinforcement learning research (reviewed by
Urbanowicz and Moore, 2009), but genetic algorithms—which we do not consider to be
reinforcement learning systems by themselves—have received much more attention, as
have other approaches to evolutionary computation (e.g., Fogel, Owens and Walsh, 1966;
Koza, 1992).
The individual most responsible for reviving the trial-and-error thread of reinforcement
learning within artificial intelligence was Harry Klopf (1972, 1975, 1982). Klopf recognizedthat essential aspects of adaptive behavior were being lost as learning researchers cameto focus almost exclusively on supervised learning. What was missing, according toKlopf, were the hedonic aspects of behavior: the drive to achieve some result from theenvironment, to control the environment toward desired ends and away from undesiredends (see Section 15.9). This is the essential idea of trial-and-error learning. Klopf’s
ideas were especially influential on the authors because our assessment of them (Bartoand Sutton, 1981a) led to our appreciation of the distinction between supervised andreinforcement learning, and to our eventual focus on reinforcement learning. Much of
the early work that we and colleagues accomplished was directed toward showing thatreinforcement learning and supervised learning were indeed di↵erent (Barto, Sutton, andBrouwer, 1981; Barto and Sutton, 1981b; Barto and Anandan, 1985). Other studies
showed how reinforcement learning could address important problems in artificial neural

20

Chapter 1: Introduction

network learning, in particular, how it could produce learning algorithms for multilayer
networks (Barto, Anderson, and Sutton, 1982; Barto and Anderson, 1985; Barto, 1985,
1986; Barto and Jordan, 1987; see Section 15.10).
We turn now to the third thread to the history of reinforcement learning, that concerning
temporal-di↵erence learning. Temporal-di↵erence learning methods are distinctive inbeing driven by the di↵erence between temporally successive estimates of the same
quantity—for example, of the probability of winning in the tic-tac-toe example. This
thread is smaller and less distinct than the other two, but it has played a particularly
important role in the field, in part because temporal-di↵erence methods seem to be new
and unique to reinforcement learning.
The origins of temporal-di↵erence learning are in part in animal learning psychology,
in particular, in the notion of secondary reinforcers. A secondary reinforcer is a stimulusthat has been paired with a primary reinforcer such as food or pain and, as a result, hascome to take on similar reinforcing properties. Minsky (1954) may have been the first to
realize that this psychological principle could be important for artificial learning systems.
Arthur Samuel (1959) was the first to propose and implement a learning method that
included temporal-di↵erence ideas, as part of his celebrated checkers-playing program(Section 16.2).
Samuel made no reference to Minsky’s work or to possible connections to animallearning. His inspiration apparently came from Claude Shannon’s (1950) suggestion thata computer could be programmed to use an evaluation function to play chess, and that it
might be able to improve its play by modifying this function online. (It is possible thatthese ideas of Shannon’s also influenced Bellman, but we know of no evidence for this.)
Minsky (1961) extensively discussed Samuel’s work in his “Steps” paper, suggesting theconnection to secondary reinforcement theories, both natural and artificial.
As we have discussed, in the decade following the work of Minsky and Samuel, little
computational work was done on trial-and-error learning, and apparently no computational
work at all was done on temporal-di↵erence learning. In 1972, Klopf brought trial-anderror learning together with an important component of temporal-di↵erence learning.
Klopf was interested in principles that would scale to learning in large systems, and thus
was intrigued by notions of local reinforcement, whereby subcomponents of an overall
learning system could reinforce one another. He developed the idea of “generalized
reinforcement,” whereby every component (nominally, every neuron) views all of its
inputs in reinforcement terms: excitatory inputs as rewards and inhibitory inputs aspunishments. This is not the same idea as what we now know as temporal-di↵erence
learning, and in retrospect it is farther from it than was Samuel’s work. On the other
hand, Klopf linked the idea with trial-and-error learning and related it to the massive
empirical database of animal learning psychology.
Sutton (1978a,b,c) developed Klopf’s ideas further, particularly the links to animallearning theories, describing learning rules driven by changes in temporally successive
predictions. He and Barto refined these ideas and developed a psychological model ofclassical conditioning based on temporal-di↵erence learning (Sutton and Barto, 1981a;
Barto and Sutton, 1982). There followed several other influential psychological models ofclassical conditioning based on temporal-di↵erence learning (e.g., Klopf, 1988; Moore et al.,1.7. Early History of Reinforcement Learning21

1986; Sutton and Barto, 1987, 1990). Some neuroscience models developed at this time
are well interpreted in terms of temporal-di↵erence learning (Hawkins and Kandel, 1984;
Byrne, Gingrich, and Baxter, 1990; Gelperin, Hopfield, and Tank, 1985; Tesauro, 1986;
Friston et al., 1994), although in most cases there was no historical connection.
Our early work on temporal-di↵erence learning was strongly influenced by animallearning theories and by Klopf’s work. Relationships to Minsky’s “Steps” paper and to
Samuel’s checkers players were recognized only afterward. By 1981, however, we were
fully aware of all the prior work mentioned above as part of the temporal-di↵erence andtrial-and-error threads. At this time we developed a method for using temporal-di↵erence
learning combined with trial-and-error learning, known as the actor–critic architecture,
and applied this method to Michie and Chambers’s pole-balancing problem (Barto,
Sutton, and Anderson, 1983). This method was extensively studied in Sutton’s (1984)
PhD dissertation and extended to use backpropagation neural networks in Anderson’s(1986) PhD dissertation. Around this time, Holland (1986) incorporated temporaldi↵erence ideas explicitly into his classifier systems in the form of his bucket-brigade
algorithm. A key step was taken by Sutton (1988) by separating temporal-di↵erence
learning from control, treating it as a general prediction method. That paper also
introduced the TD( ) algorithm and proved some of its convergence properties.
As we were finalizing our work on the actor–critic architecture in 1981, we discovereda paper by Ian Witten (1977, 1976a) which appears to be the earliest publication of atemporal-di↵erence learning rule. He proposed the method that we now call tabular TD(0)for use as part of an adaptive controller for solving MDPs. This work was first submitted
for journal publication in 1974 and also appeared in Witten’s 1976 PhD dissertation.
Witten’s work was a descendant of Andreae’s early experiments with STeLLA and other
trial-and-error learning systems. Thus, Witten’s 1977 paper spanned both major threadsof reinforcement learning research—trial-and-error learning and optimal control—while
making a distinct early contribution to temporal-di↵erence learning.
The temporal-di↵erence and optimal control threads were fully brought together
in 1989 with Chris Watkins’s development of Q-learning. This work extended andintegrated prior work in all three threads of reinforcement learning research. Paul Werbos(1987) contributed to this integration by arguing for the convergence of trial-and-error
learning and dynamic programming since 1977. By the time of Watkins’s work there had
been tremendous growth in reinforcement learning research, primarily in the machinelearning subfield of artificial intelligence, but also in artificial neural networks and artificial
intelligence more broadly. In 1992, the remarkable success of Gerry Tesauro’s backgammon
playing program, TD-Gammon, brought additional attention to the field.
In the time since publication of the first edition of this book, a flourishing subfield of
neuroscience developed that focuses on the relationship between reinforcement learningalgorithms and reinforcement learning in the nervous system. Most responsible for this isan uncanny similarity between the behavior of temporal-di↵erence algorithms and theactivity of dopamine producing neurons in the brain, as pointed out by a number of
researchers (Friston et al., 1994; Barto, 1995a; Houk, Adams, and Barto, 1995; Montague,
Dayan, and Sejnowski, 1996; and Schultz, Dayan, and Montague, 1997). Chapter 15
provides an introduction to this exciting aspect of reinforcement learning. Other important

22

Chapter 1: Introduction

contributions made in the recent history of reinforcement learning are too numerous tomention in this brief account; we cite many of these at the end of the individual chaptersin which they arise.

Bibliographical Remarks
For additional general coverage of reinforcement learning, we refer the reader to thebooks by Szepesvári (2010), Bertsekas and Tsitsiklis (1996), Kaelbling (1993a), andSugiyama, Hachiya, and Morimura (2013). Books that take a control or operations research
perspective include those of Si, Barto, Powell, and Wunsch (2004), Powell (2011), Lewis
and Liu (2012), and Bertsekas (2012). Cao’s (2009) review places reinforcement learningin the context of other approaches to learning and optimization of stochastic dynamic
systems. Three special issues of the journal Machine Learning focus on reinforcement
learning: Sutton (1992a), Kaelbling (1996), and Singh (2002). Useful surveys are provided
by Barto (1995b); Kaelbling, Littman, and Moore (1996); and Keerthi and Ravindran(1997). The volume edited by Weiring and van Otterlo (2012) provides an excellent
overview of recent developments.
1.2

The example of Phil’s breakfast in this chapter was inspired by Agre (1988).

1.5

The temporal-di↵erence method used in the tic-tac-toe example is developed in
Chapter 6.

Part I: Tabular Solution Methods

In this part of the book we describe almost all the core ideas of reinforcement learningalgorithms in their simplest forms: that in which the state and action spaces are smallenough for the approximate value functions to be represented as arrays, or tables. In
this case, the methods can often find exact solutions, that is, they can often find exactly
the optimal value function and the optimal policy. This contrasts with the approximate
methods described in the next part of the book, which only find approximate solutions,but which in return can be applied e↵ectively to much larger problems.
The first chapter of this part of the book describes solution methods for the special
case of the reinforcement learning problem in which there is only a single state, calledbandit problems. The second chapter describes the general problem formulation that wetreat throughout the rest of the book—finite Markov decision processes—and its mainideas including Bellman equations and value functions.
The next three chapters describe three fundamental classes of methods for solving finite
Markov decision problems: dynamic programming, Monte Carlo methods, and temporaldi↵erence learning. Each class of methods has its strengths and weaknesses. Dynamic
programming methods are well developed mathematically, but require a complete andaccurate model of the environment. Monte Carlo methods don’t require a model and areconceptually simple, but are not well suited for step-by-step incremental computation.
Finally, temporal-di↵erence methods require no model and are fully incremental, but are
more complex to analyze. The methods also di↵er in several ways with respect to theirefficiency and speed of convergence.
The remaining two chapters describe how these three classes of methods can be
combined to obtain the best features of each of them. In one chapter we describe how
the strengths of Monte Carlo methods can be combined with the strengths of temporaldi↵erence methods via multi-step bootstrapping methods. In the final chapter of this partof the book we show how temporal-di↵erence learning methods can be combined withmodel learning and planning methods (such as dynamic programming) for a complete
and unified solution to the tabular reinforcement learning problem.

Chapter 2

Multi-armed Bandits
The most important feature distinguishing reinforcement learning from other types oflearning is that it uses training information that evaluates the actions taken rather
than instructs by giving correct actions. This is what creates the need for active
exploration, for an explicit search for good behavior. Purely evaluative feedback indicates
how good the action taken was, but not whether it was the best or the worst action
possible. Purely instructive feedback, on the other hand, indicates the correct action totake, independently of the action actually taken. This kind of feedback is the basis ofsupervised learning, which includes large parts of pattern classification, artificial neuralnetworks, and system identification. In their pure forms, these two kinds of feedback
are quite distinct: evaluative feedback depends entirely on the action taken, whereas
instructive feedback is independent of the action taken.
In this chapter we study the evaluative aspect of reinforcement learning in a simplified
setting, one that does not involve learning to act in more than one situation. Thisnonassociative setting is the one in which most prior work involving evaluative feedback
has been done, and it avoids much of the complexity of the full reinforcement learning
problem. Studying this case enables us to see most clearly how evaluative feedback di↵ers
from, and yet can be combined with, instructive feedback.
The particular nonassociative, evaluative feedback problem that we explore is a simple
version of the k -armed bandit problem. We use this problem to introduce a numberof basic learning methods which we extend in later chapters to apply to the full reinforcement learning problem. At the end of this chapter, we take a step closer to the full
reinforcement learning problem by discussing what happens when the bandit problem
becomes associative, that is, when the best action depends on the situation.

2.1

A k -armed Bandit Problem

Consider the following learning problem. You are faced repeatedly with a choice amongk di↵erent options, or actions. After each choice you receive a numerical reward chosenfrom a stationary probability distribution that depends on the action you selected. Your

26

Chapter 2: Multi-armed Bandits

objective is to maximize the expected total reward over some time period, for example,
over 1000 action selections, or time steps.
This is the original form of the k-armed bandit problem, so named by analogy to a slot
machine, or “one-armed bandit,” except that it has k levers instead of one. Each action
selection is like a play of one of the slot machine’s levers, and the rewards are the payo↵sfor hitting the jackpot. Through repeated action selections you are to maximize yourwinnings by concentrating your actions on the best levers. Another analogy is that ofa doctor choosing between experimental treatments for a series of seriously ill patients.
Each action is the selection of a treatment, and each reward is the survival or well-beingof the patient. Today the term “bandit problem” is sometimes used for a generalizationof the problem described above, but in this book we use it to refer just to this simple
case.
In our k -armed bandit problem, each of the k actions has an expected or mean reward
given that that action is selected; let us call this the value of that action. We denote theaction selected on time step t as At , and the corresponding reward as Rt . The value thenof an arbitrary action a, denoted q⇤ (a), is the expected reward given that a is selected:.
q⇤ (a) = E[Rt | At = a] .
If you knew the value of each action, then it would be trivial to solve the k -armed bandit
problem: you would always select the action with highest value. We assume that you do
not know the action values with certainty, although you may have estimates. We denotethe estimated value of action a at time step t as Qt (a). We would like Qt (a) to be closeto q⇤ (a).
If you maintain estimates of the action values, then at any time step there is at least
one action whose estimated value is greatest. We call these the greedy actions. When you
select one of these actions, we say that you are exploiting your current knowledge of thevalues of the actions. If instead you select one of the nongreedy actions, then we say you
are exploring, because this enables you to improve your estimate of the nongreedy action’svalue. Exploitation is the right thing to do to maximize the expected reward on the one
step, but exploration may produce the greater total reward in the long run. For example,
suppose a greedy action’s value is known with certainty, while several other actions are
estimated to be nearly as good but with substantial uncertainty. The uncertainty issuch that at least one of these other actions probably is actually better than the greedy
action, but you don’t know which one. If you have many time steps ahead on which
to make action selections, then it may be better to explore the nongreedy actions anddiscover which of them are better than the greedy action. Reward is lower in the shortrun, during exploration, but higher in the long run because after you have discovered
the better actions, you can exploit them many times. Because it is not possible both to
explore and to exploit with any single action selection, one often refers to the “conflict”
between exploration and exploitation.
In any specific case, whether it is better to explore or exploit depends in a complex
way on the precise values of the estimates, uncertainties, and the number of remaining
steps. There are many sophisticated methods for balancing exploration and exploitation
for particular mathematical formulations of the k -armed bandit and related problems.

2.2. Action-value Methods

27

However, most of these methods make strong assumptions about stationarity and priorknowledge that are either violated or impossible to verify in most applications and in
the full reinforcement learning problem that we consider in subsequent chapters. The
guarantees of optimality or bounded loss for these methods are of little comfort when theassumptions of their theory do not apply.
In this book we do not worry about balancing exploration and exploitation in a
sophisticated way; we worry only about balancing them at all. In this chapter we present
several simple balancing methods for the k -armed bandit problem and show that theywork much better than methods that always exploit. The need to balance explorationand exploitation is a distinctive challenge that arises in reinforcement learning; thesimplicity of our version of the k -armed bandit problem enables us to show this in aparticularly clear form.

2.2

Action-value Methods

We begin by looking more closely at methods for estimating the values of actions andfor using the estimates to make action selection decisions, which we collectively callaction-value methods. Recall that the true value of an action is the mean reward whenthat action is selected. One natural way to estimate this is by averaging the rewards
actually received:. sum of rewards when a taken prior to t
Qt (a) ==number of times a taken prior to t

Pt 1

i=1 Ri ·
P
t 1
i=1

Ai =a

,

(2.1)

Ai =a

where predicate denotes the random variable that is 1 if predicate is true and 0 if it is not.
If the denominator is zero, then we instead define Qt (a) as some default value, such as
0. As the denominator goes to infinity, by the law of large numbers, Qt (a) converges toq⇤ (a). We call this the sample-average method for estimating action values because eachestimate is an average of the sample of relevant rewards. Of course this is just one way
to estimate action values, and not necessarily the best one. Nevertheless, for now let us
stay with this simple estimation method and turn to the question of how the estimatesmight be used to select actions.
The simplest action selection rule is to select one of the actions with the highest
estimated value, that is, one of the greedy actions as defined in the previous section.
If there is more than one greedy action, then a selection is made among them in somearbitrary way, perhaps randomly. We write this greedy action selection method as.
At = argmax Qt (a),

(2.2)

a

where argmaxa denotes the action a for which the expression that follows is maximized(with ties broken arbitrarily). Greedy action selection always exploits current knowledge tomaximize immediate reward; it spends no time at all sampling apparently inferior actions
to see if they might really be better. A simple alternative is to behave greedily most of
the time, but every once in a while, say with small probability ", instead select randomly

28

Chapter 2: Multi-armed Bandits

from among all the actions with equal probability, independently of the action-value
estimates. We call methods using this near-greedy action selection rule "-greedy methods.
An advantage of these methods is that, in the limit as the number of steps increases,
every action will be sampled an infinite number of times, thus ensuring that all the Qt (a)converge to their respective q⇤ (a). This of course implies that the probability of selecting
the optimal action converges to greater than 1 ", that is, to near certainty. These arejust asymptotic guarantees, however, and say little about the practical e↵ectiveness ofthe methods.
Exercise 2.1 In "-greedy action selection, for the case of two actions and " = 0.5, what isthe probability that the greedy action is selected?
⇤

2.3

The 10-armed Testbed

To roughly assess the relative e↵ectiveness of the greedy and "-greedy action-value
methods, we compared them numerically on a suite of test problems. This was a setof 2000 randomly generated k -armed bandit problems with k = 10. For each bandit
problem, such as the one shown in Figure 2.1, the action values, q⇤ (a), a = 1, . . . , 10,32q⇤ (3)q⇤ (5)

1

Rewarddistribution0q⇤ (9)q⇤ (4)q⇤ (1)q⇤ (7)q⇤ (10)q⇤ (2)-1q⇤ (8)q⇤ (6)

-2
-3

1

2

3

4

5

6

7

8

9

10

ActionFigure 2.1: An example bandit problem from the 10-armed testbed. The true value q⇤ (a) ofeach of the ten actions was selected according to a normal distribution with mean zero and unit
variance, and then the actual rewards were selected according to a mean q⇤ (a), unit-variancenormal distribution, as suggested by these gray distributions.

2.3. The 10-armed Testbed

29

were selected according to a normal (Gaussian) distribution with mean 0 and variance 1.
Then, when a learning method applied to that problem selected action At at time step t,
the actual reward, Rt , was selected from a normal distribution with mean q⇤ (At ) andvariance 1. These distributions are shown in gray in Figure 2.1. We call this suite of test
tasks the 10-armed testbed. For any learning method, we can measure its performanceand behavior as it improves with experience over 1000 time steps when applied to one of
the bandit problems. This makes up one run. Repeating this for 2000 independent runs,
each with a di↵erent bandit problem, we obtained measures of the learning algorithm’s
average behavior.
Figure 2.2 compares a greedy method with two "-greedy methods (" = 0.01 and " = 0.1),
as described above, on the 10-armed testbed. All the methods formed their action-value
estimates using the sample-average technique (with an initial estimate of 0). The uppergraph shows the increase in expected reward with experience. The greedy method
improved slightly faster than the other methods at the very beginning, but then leveledo↵ at a lower level. It achieved a reward-per-step of only about 1, compared with the best
possible of about 1.54 on this testbed. The greedy method performed significantly worsein the long run because it often got stuck performing suboptimal actions. The lower graph
1.5" = 0.1" = 0.011" = 0 (greedy)

Averagereward
0.5

0
01

250

500

750

1000

Steps
100%
80%

" = 0.1
60%%
Optimalaction 40%" = 0.01" = 0 (greedy)

20%
0%
01

250

500

750

1000

Steps

Figure 2.2: Average performance of "-greedy action-value methods on the 10-armed testbed.
These data are averages over 2000 runs with di↵erent bandit problems. All methods used sample
averages as their action-value estimates.

30

Chapter 2: Multi-armed Bandits

shows that the greedy method found the optimal action in only approximately one-thirdof the tasks. In the other two-thirds, its initial samples of the optimal action weredisappointing, and it never returned to it. The "-greedy methods eventually performed
better because they continued to explore and to improve their chances of recognizing
the optimal action. The " = 0.1 method explored more, and usually found the optimal
action earlier, but it never selected that action more than 91% of the time. The " = 0.01
method improved more slowly, but eventually would perform better than the " = 0.1
method on both performance measures shown in the figure. It is also possible to reduce "over time to try to get the best of both high and low values.
The advantage of "-greedy over greedy methods depends on the task. For example,
suppose the reward variance had been larger, say 10 instead of 1. With noisier rewards
it takes more exploration to find the optimal action, and "-greedy methods should fareeven better relative to the greedy method. On the other hand, if the reward varianceswere zero, then the greedy method would know the true value of each action after trying
it once. In this case the greedy method might actually perform best because it would
soon find the optimal action and then never explore. But even in the deterministic casethere is a large advantage to exploring if we weaken some of the other assumptions. For
example, suppose the bandit task were nonstationary, that is, the true values of theactions changed over time. In this case exploration is needed even in the deterministic
case to make sure one of the nongreedy actions has not changed to become better than
the greedy one. As we shall see in the next few chapters, nonstationarity is the casemost commonly encountered in reinforcement learning. Even if the underlying task isstationary and deterministic, the learner faces a set of banditlike decision tasks each ofwhich changes over time as learning proceeds and the agent’s decision-making policychanges. Reinforcement learning requires a balance between exploration and exploitation.
Exercise 2.2: Bandit example Consider a k -armed bandit problem with k = 4 actions,denoted 1, 2, 3, and 4. Consider applying to this problem a bandit algorithm using"-greedy action selection, sample-average action-value estimates, and initial estimatesof Q1 (a) = 0, for all a. Suppose the initial sequence of actions and rewards is A1 = 1,
R1 = 1, A2 = 2, R2 = 1, A3 = 2, R3 = 2, A4 = 2, R4 = 2, A5 = 3, R5 = 0. On someof these time steps the " case may have occurred, causing an action to be selected at
random. On which time steps did this definitely occur? On which time steps could thispossibly have occurred?
⇤
Exercise 2.3 In the comparison shown in Figure 2.2, which method will perform best in
the long run in terms of cumulative reward and probability of selecting the best action?
How much better will it be? Express your answer quantitatively.
⇤

2.4

Incremental Implementation

The action-value methods we have discussed so far all estimate action values as sample
averages of observed rewards. We now turn to the question of how these averages can becomputed in a computationally efficient manner, in particular, with constant memoryand constant per-time-step computation.

2.4. Incremental Implementation

31

To simplify notation we concentrate on a single action. Let Ri now denote the reward
received after the ith selection of this action, and let Qn denote the estimate of its action
value after it has been selected n 1 times, which we can now write simply as. R1 + R2 + · · · + Rn 1
Qn =
.
n 1
The obvious implementation would be to maintain a record of all the rewards and thenperform this computation whenever the estimated value was needed. However, if this isdone, then the memory and computational requirements would grow over time as more
rewards are seen. Each additional reward would require additional memory to store it
and additional computation to compute the sum in the numerator.
As you might suspect, this is not really necessary. It is easy to devise incremental
formulas for updating averages with small, constant computation required to processeach new reward. Given Qn and the nth reward, Rn , the new average of all n rewards
can be computed by
n

Qn+1

=

1X
Ri
n i=1

=

1
n

=
=
=
=

1
n

Rn +

n
X1
i=1

Rn + (n

Ri

!

1)

1
n

⌘
1⇣
Rn + (n 1)Qnn⌘1⇣
Rn + nQn Qn
n
i
1h
Qn +
Rn Q n ,
n

n
X1

1 i=1

Ri

!

(2.3)

which holds even for n = 1, obtaining Q2 = R1 for arbitrary Q1 . This implementation
requires memory only for Qn and n, and only the small computation (2.3) for each new
reward.
This update rule (2.3) is of a form that occurs frequently throughout this book. The
general form ishi
NewEstimate
OldEstimate + StepSize Target OldEstimate .
(2.4)

⇥
⇤
The expression Target OldEstimate is an error in the estimate. It is reduced by taking
a step toward the “Target.” The target is presumed to indicate a desirable direction inwhich to move, though it may be noisy. In the case above, for example, the target is thenth reward.
Note that the step-size parameter (StepSize) used in the incremental method (2.3)
changes from time step to time step. In processing the nth reward for action a, the32

Chapter 2: Multi-armed Bandits

method uses the step-size parameter n1 . In this book we denote the step-size parameterby ↵ or, more generally, by ↵t (a).
Pseudocode for a complete bandit algorithm using incrementally computed sample
averages and "-greedy action selection is shown in the box below. The function bandit(a)is assumed to take an action and return a corresponding reward.
A simple bandit algorithm
Initialize, for a = 1 to k:
Q(a)
0
N (a)
0
Loop forever:⇢argmaxa Q(a)with probability 1A
a random action with probability "Rbandit(A)N (A)
N (A) + 1 ⇥⇤1
Q(A)
Q(A) + N (A)R Q(A)

2.5"(breaking ties randomly)

Tracking a Nonstationary Problem

The averaging methods discussed so far are appropriate for stationary bandit problems,that is, for bandit problems in which the reward probabilities do not change over time.
As noted earlier, we often encounter reinforcement learning problems that are e↵ectively
nonstationary. In such cases it makes sense to give more weight to recent rewards thanto long-past rewards. One of the most popular ways of doing this is to use a constant
step-size parameter. For example, the incremental update rule (2.3) for updating an
average Qn of the n 1 past rewards is modified to behi.
Qn+1 = Qn + ↵ Rn Qn ,(2.5)
where the step-size parameter ↵ 2 (0, 1] is constant. This results in Qn+1 being a weighted
average of past rewards and the initial estimate Q1 :hiQn+1 = Qn + ↵ Rn Qn=↵Rn + (1↵)Qn=

↵Rn + (1↵) [↵Rn 1 + (1↵)Qn 1 ]=↵Rn + (1↵)↵Rn 1 + (1↵)2 Qn 1=↵Rn + (1↵)↵Rn 1 + (1↵)2 ↵Rn 2 +=(1· · · + (1 ↵)n 1 ↵R1 + (1nX↵)n Q1 +↵(1 ↵)n i Ri .
i=1↵)n Q1(2.6)

2.5. Tracking a Nonstationary Problem

33Pn
We call this a weighted average because the sum of the weights is (1 ↵)n + i=1 ↵(1↵)n i = 1, as you can check for yourself. Note that the weight, ↵(1 ↵)n i , given to thereward Ri depends on how many rewards ago, n i, it was observed. The quantity 1 ↵is less than 1, and thus the weight given to Ri decreases as the number of intervening
rewards increases. In fact, the weight decays exponentially according to the exponent
on 1 ↵. (If 1 ↵ = 0, then all the weight goes on the very last reward, Rn , becauseof the convention that 00 = 1.) Accordingly, this is sometimes called an exponential
recency-weighted average.
Sometimes it is convenient to vary the step-size parameter from step to step. Let ↵n (a)denote the step-size parameter used to process the reward received after the nth selectionof action a. As we have noted, the choice ↵n (a) = n1 results in the sample-average method,which is guaranteed to converge to the true action values by the law of large numbers.
But of course convergence is not guaranteed for all choices of the sequence {↵n (a)}. A
well-known result in stochastic approximation theory gives us the conditions required toassure convergence with probability 1:
1
X

n=1

↵n (a) = 1

and

1
X

n=1

↵n2 (a) < 1.

(2.7)

The first condition is required to guarantee that the steps are large enough to eventually
overcome any initial conditions or random fluctuations. The second condition guarantees
that eventually the steps become small enough to assure convergence.
Note that both convergence conditions are met for the sample-average case, ↵n (a) = n1 ,but not for the case of constant step-size parameter, ↵n (a) = ↵. In the latter case, thesecond condition is not met, indicating that the estimates never completely converge but
continue to vary in response to the most recently received rewards. As we mentioned
above, this is actually desirable in a nonstationary environment, and problems that aree↵ectively nonstationary are the most common in reinforcement learning. In addition,
sequences of step-size parameters that meet the conditions (2.7) often converge very slowly
or need considerable tuning in order to obtain a satisfactory convergence rate. Althoughsequences of step-size parameters that meet these convergence conditions are often usedin theoretical work, they are seldom used in applications and empirical research.
Exercise 2.4 If the step-size parameters, ↵n , are not constant, then the estimate Qn isa weighted average of previously received rewards with a weighting di↵erent from thatgiven by (2.6). What is the weighting on each prior reward for the general case, analogous
to (2.6), in terms of the sequence of step-size parameters?
⇤
Exercise 2.5 (programming) Design and conduct an experiment to demonstrate thedifficulties that sample-average methods have for nonstationary problems. Use a modified
version of the 10-armed testbed in which all the q⇤ (a) start out equal and then takeindependent random walks (say by adding a normally distributed increment with mean 0
and standard deviation 0.01 to all the q⇤ (a) on each step). Prepare plots like Figure 2.2
for an action-value method using sample averages, incrementally computed, and anotheraction-value method using a constant step-size parameter, ↵ = 0.1. Use " = 0.1 andlonger runs, say of 10,000 steps.
⇤

34

2.6

Chapter 2: Multi-armed Bandits

Optimistic Initial Values

All the methods we have discussed so far are dependent to some extent on the initialaction-value estimates, Q1 (a). In the language of statistics, these methods are biasedby their initial estimates. For the sample-average methods, the bias disappears once all
actions have been selected at least once, but for methods with constant ↵, the bias ispermanent, though decreasing over time as given by (2.6). In practice, this kind of bias
is usually not a problem and can sometimes be very helpful. The downside is that theinitial estimates become, in e↵ect, a set of parameters that must be picked by the user, if
only to set them all to zero. The upside is that they provide an easy way to supply someprior knowledge about what level of rewards can be expected.
Initial action values can also be used as a simple way to encourage exploration. Supposethat instead of setting the initial action values to zero, as we did in the 10-armed testbed,
we set them all to +5. Recall that the q⇤ (a) in this problem are selected from a normal
distribution with mean 0 and variance 1. An initial estimate of +5 is thus wildly optimistic.
But this optimism encourages action-value methods to explore. Whichever actions are
initially selected, the reward is less than the starting estimates; the learner switches toother actions, being “disappointed” with the rewards it is receiving. The result is that all
actions are tried several times before the value estimates converge. The system does afair amount of exploration even if greedy actions are selected all the time.
Figure 2.3 shows the performance on the 10-armed bandit testbed of a greedy method
using Q1 (a) = +5, for all a. For comparison, also shown is an "-greedy method withQ1 (a) = 0. Initially, the optimistic method performs worse because it explores more,
but eventually it performs better because its exploration decreases with time. We callthis technique for encouraging exploration optimistic initial values. We regard it asa simple trick that can be quite e↵ective on stationary problems, but it is far frombeing a generally useful approach to encouraging exploration. For example, it is notwell suited to nonstationary problems because its drive for exploration is inherently

100%

Optimistic,
greedyoptimistic,
greedyQ1 = 5, " = 0
Q01 = 5, = 080%" -greedy
Realistic,
realistic, -greedy
Q1 = 0, " = 0.1

60%%
Optimalaction 40%

Q01 = 0, = 0.1

20%
0%

01

200

400

600

800

1000

Plays
Steps
Figure 2.3: The e↵ect of optimistic initial action-value estimates on the 10-armed testbed.
Both methods used a constant step-size parameter, ↵ = 0.1.

2.7. Upper-Confidence-Bound Action Selection

35

temporary. If the task changes, creating a renewed need for exploration, this method
cannot help. Indeed, any method that focuses on the initial conditions in any special way
is unlikely to help with the general nonstationary case. The beginning of time occurs
only once, and thus we should not focus on it too much. This criticism applies as well tothe sample-average methods, which also treat the beginning of time as a special event,averaging all subsequent rewards with equal weights. Nevertheless, all of these methods
are very simple, and one of them—or some simple combination of them—is often adequate
in practice. In the rest of this book we make frequent use of several of these simple
exploration techniques.
Exercise 2.6: Mysterious Spikes The results shown in Figure 2.3 should be quite reliablebecause they are averages over 2000 individual, randomly chosen 10-armed bandit tasks.
Why, then, are there oscillations and spikes in the early part of the curve for the optimistic
method? In other words, what might make this method perform particularly better orworse, on average, on particular early steps?
⇤
Exercise 2.7: Unbiased Constant-Step-Size Trick In most of this chapter we have usedsample averages to estimate action values because sample averages do not produce theinitial bias that constant step sizes do (see the analysis leading to (2.6)). However, sample
averages are not a completely satisfactory solution because they may perform poorly
on nonstationary problems. Is it possible to avoid the bias of constant step sizes while
retaining their advantages on nonstationary problems? One way is to use a step size of.
(2.8)
n = ↵/ōn ,to process the nth reward for a particular action, where ↵ > 0 is a conventional constantstep size, and ōn is a trace of one that starts at 0:.
.
ōn = ōn 1 + ↵(1 ōn 1 ), for n > 0, with ō0 = 0.
(2.9)
Carry out an analysis like that in (2.6) to show that Qn is an exponential recency-weighted
average without initial bias.
⇤

2.7

Upper-Confidence-Bound Action Selection

Exploration is needed because there is always uncertainty about the accuracy of theaction-value estimates. The greedy actions are those that look best at present, but some ofthe other actions may actually be better. "-greedy action selection forces the non-greedy
actions to be tried, but indiscriminately, with no preference for those that are nearly
greedy or particularly uncertain. It would be better to select among the non-greedy
actions according to their potential for actually being optimal, taking into account both
how close their estimates are to being maximal and the uncertainties in those estimates.
One e↵ective way of doing this is to select actions according to
s
"
#
ln t
.
At = argmax Qt (a) + c,(2.10)
Nt (a)awhere ln t denotes the natural logarithm of t (the number that e ⇡ 2.71828 would haveto be raised to in order to equal t), Nt (a) denotes the number of times that action a has

36

Chapter 2: Multi-armed Bandits

been selected prior to time t (the denominator in (2.1)), and the number c > 0 controls
the degree of exploration. If Nt (a) = 0, then a is considered to be a maximizing action.
The idea of this upper confidence bound (UCB) action selection is that the square-root
term is a measure of the uncertainty or variance in the estimate of a’s value. The quantity
being max’ed over is thus a sort of upper bound on the possible true value of action a, withc determining the confidence level. Each time a is selected the uncertainty is presumably
reduced: Nt (a) increments, and, as it appears in the denominator, the uncertainty term
decreases. On the other hand, each time an action other than a is selected, t increases but
Nt (a) does not; because t appears in the numerator, the uncertainty estimate increases.
The use of the natural logarithm means that the increases get smaller over time, but are
unbounded; all actions will eventually be selected, but actions with lower value estimates,or that have already been selected frequently, will be selected with decreasing frequency
over time.
Results with UCB on the 10-armed testbed are shown in Figure 2.4. UCB often
performs well, as shown here, but is more difficult than "-greedy to extend beyond bandits
to the more general reinforcement learning settings considered in the rest of this book.
One difficulty is in dealing with nonstationary problems; methods more complex thanthose presented in Section 2.5 would be needed. Another difficulty is dealing with large
state spaces, particularly when using function approximation as developed in Part II ofthis book. In these more advanced settings the idea of UCB action selection is usually
not practical.
UCB c = 2

1.5

-greedy= 0.1

1

Averagereward
0.5

0
1

250

500

750

1000

Steps
Figure 2.4: Average performance of UCB action selection on the 10-armed testbed. As shown,
UCB generally performs better than "-greedy action selection, except in the first k steps, when
it selects randomly among the as-yet-untried actions.

Exercise 2.8: UCB Spikes In Figure 2.4 the UCB algorithm shows a distinct spikein performance on the 11th step. Why is this? Note that for your answer to be fully
satisfactory it must explain both why the reward increases on the 11th step and why it
decreases on the subsequent steps. Hint: If c = 1, then the spike is less prominent. ⇤2.8. Gradient Bandit Algorithms

2.8

37

Gradient Bandit Algorithms

So far in this chapter we have considered methods that estimate action values and use
those estimates to select actions. This is often a good approach, but it is not the only
one possible. In this section we consider learning a numerical preference for each actiona, which we denote Ht (a) 2 R. The larger the preference, the more often that action istaken, but the preference has no interpretation in terms of reward. Only the relative
preference of one action over another is important; if we add 1000 to all the action
preferences there is no e↵ect on the action probabilities, which are determined according
to a soft-max distribution (i.e., Gibbs or Boltzmann distribution) as follows:eHt (a).
.
Pr{At = a} = Pk
= ⇡t (a),
H
(b)
t
b=1 e

(2.11)

where here we have also introduced a useful new notation, ⇡t (a), for the probability of
taking action a at time t. Initially all action preferences are the same (e.g., H1 (a) = 0,
for all a) so that all actions have an equal probability of being selected.
Exercise 2.9 Show that in the case of two actions, the soft-max distribution is the sameas that given by the logistic, or sigmoid, function often used in statistics and artificial
neural networks.
⇤
There is a natural learning algorithm for soft-max action preferences based on the ideaof stochastic gradient ascent. On each step, after selecting action At and receiving the
reward Rt , the action preferences are updated by:.
Ht+1 (At ) = Ht (At ) + ↵ Rt R̄t 1 ⇡t (At ) ,.
Ht+1 (a) = Ht (a) ↵ Rt R̄t ⇡t (a),andfor all a 6= At ,(2.12)

where ↵ > 0 is a step-size parameter, and R̄t 2 R is the average of the rewards up to but.
not including time t (with R̄1 = R1 ), which can be computed incrementally as describedin Section 2.4 (or Section 2.5 if the problem is nonstationary).1 The R̄t term serves as abaseline with which the reward is compared. If the reward is higher than the baseline,
then the probability of taking At in the future is increased, and if the reward is below
baseline, then the probability is decreased. The non-selected actions move in the opposite
direction.
Figure 2.5 shows results with the gradient bandit algorithm on a variant of the 10armed testbed in which the true expected rewards were selected according to a normal
distribution with a mean of +4 instead of zero (and with unit variance as before). This
shifting up of all the rewards has absolutely no e↵ect on the gradient bandit algorithm
because of the reward baseline term, which instantaneously adapts to the new level. But
if the baseline were omitted (that is, if R̄t was taken to be constant zero in (2.12)), thenperformance would be significantly degraded, as shown in the figure.
1 In the empirical results in this chapter, the baseline R̄t also included Rt .

38

Chapter 2: Multi-armed Bandits100%

α = 0.1

80%

with baseline

α = 0.4%
Optimalaction60%

α = 0.1
40%

without baseline

α = 0.4

20%
0%
1

250

500

750

1000

Steps
Figure 2.5: Average performance of the gradient bandit algorithm with and without a rewardbaseline on the 10-armed testbed when the q⇤ (a) are chosen to be near +4 rather than near zero.

The Bandit Gradient Algorithm as Stochastic Gradient Ascent
One can gain a deeper insight into the gradient bandit algorithm by understanding
it as a stochastic approximation to gradient ascent. In exact gradient ascent, eachaction preference Ht (a) would be incremented in proportion to the increment’se↵ect on performance:@ E [Rt ].
Ht+1 (a) = Ht (a) + ↵,@Ht (a)(2.13)

where the measure of performance here is the expected reward:
X
E[Rt ] =⇡t (x)q⇤ (x),xand the measure of the increment’s e↵ect is the partial derivative of this performancemeasure with respect to the action preference. Of course, it is not possible toimplement gradient ascent exactly in our case because by assumption we do notknow the q⇤ (x), but in fact the updates of our algorithm (2.12) are equal to (2.13)
in expected value, making the algorithm an instance of stochastic gradient ascent.
The calculations showing this require only beginning calculus, but take several

2.8. Gradient Bandit Algorithms

39

steps. First we take a closer look at the exact performance gradient:"#X@ E[Rt ]@=⇡t (x)q⇤ (x)@Ht (a)@Ht (a) xX@ ⇡t (x)=q⇤ (x)@Ht (a)xX@ ⇡t (x)=q⇤ (x) Bt
,
@H
t (a)
x
where Bt , called the baseline, can be any scalar that does not depend on x. We can
include a baseline here without changing the equality because the gradient sums
P @ ⇡t (x)to zero over all the actions, x @H
= 0. As Ht (a) is changed, some actions’t (a)probabilities go up and some go down, but the sum of the changes must be zero
because the sum of the probabilities is always one.
Next we multiply each term of the sum by ⇡t (x)/⇡t (x):@ E[Rt ] X=⇡t (x) q⇤ (x)@Ht (a)x

Bt@ ⇡t (x)/⇡t (x).
@Ht (a)The equation is now in the form of an expectation, summing over all possible values
x of the random variable At , then multiplying by the probability of taking thosevalues. Thus:@ ⇡t (At )= E q⇤ (At ) Bt/⇡t (At )@Ht (a)@ ⇡t (At )= E Rt R̄t/⇡t (At ) ,@Ht (a)where here we have chosen the baseline Bt = R̄t and substituted Rt for q⇤ (At ),which is permitted because E[Rt |At ] = q⇤ (At ). Shortly we will establish that@ ⇡t (x)⇡t (a) , where a=x is defined to be 1 if a = x, else 0.
a=x@Ht (a) = ⇡t (x)Assuming that for now, we have⇥⇤
= E Rt R̄t ⇡t (At ) a=At ⇡t (a) /⇡t (At )⇥⇤= E Rt R̄t⇡t (a) .
a=At
Recall that our plan has been to write the performance gradient as an expectationof something that we can sample on each step, as we have just done, and thenupdate on each step in proportion to the sample. Substituting a sample of the
expectation above for the performance gradient in (2.13) yields:
Ht+1 (a) = Ht (a) + ↵ Rt

R̄t

a=At

⇡t (a) ,for all a,which you may recognize as being equivalent to our original algorithm (2.12).

40

Chapter 2: Multi-armed Bandits@ ⇡t (x)Thus it remains only to show that @H= ⇡t (x)t (a)Recall the standard quotient rule for derivatives:

a=x⇡t (a) , as we assumed.

@f (x)
g(x) f (x) @g(x)@ f (x)@x= @x.
@x g(x)g(x)2Using this, we can write@ ⇡t (x)@=⇡t (x)@Ht (a)@Ht (a)"#@eHt (x)=Pk
Ht (y)@Ht (a)y=1 e==@eHt (x)@Ht (a)

a=x e

y=1 e

Ht (x)

a=x e

= Pk

Pk

=

Pk

Ht (y)

a=x

eHt (x)⌘2@Pk

Ht (y)y=1 e@Ht (a)k
Ht (y)
y=1 e

y=1 e

⇣P

a=x ⇡t (x)= ⇡t (x)⇣P

Ht (x)y=1 e

Ht (y)Ht (y)k
Ht (y)
y=1 e

eHt (x) eHt (a)⌘2(by the quotient rule)xx(because @e@x = e )eHt (x) eHt (a)⇣P⌘2k
Ht (y)y=1 e⇡t (x)⇡t (a)⇡t (a) .

Q.E.D.

We have just shown that the expected update of the gradient bandit algorithm
is equal to the gradient of expected reward, and thus that the algorithm is an
instance of stochastic gradient ascent. This assures us that the algorithm has robust
convergence properties.
Note that we did not require any properties of the reward baseline other thanthat it does not depend on the selected action. For example, we could have set
it to zero, or to 1000, and the algorithm would still be an instance of stochastic
gradient ascent. The choice of the baseline does not a↵ect the expected updateof the algorithm, but it does a↵ect the variance of the update and thus the rate ofconvergence (as shown, for example, in Figure 2.5). Choosing it as the average ofthe rewards may not be the very best, but it is simple and works well in practice.

2.9. Associative Search (Contextual Bandits)

2.9

41

Associative Search (Contextual Bandits)

So far in this chapter we have considered only nonassociative tasks, that is, tasks in which
there is no need to associate di↵erent actions with di↵erent situations. In these tasks
the learner either tries to find a single best action when the task is stationary, or tries totrack the best action as it changes over time when the task is nonstationary. However,in a general reinforcement learning task there is more than one situation, and the goal
is to learn a policy: a mapping from situations to the actions that are best in thosesituations. To set the stage for the full problem, we briefly discuss the simplest way inwhich nonassociative tasks extend to the associative setting.
As an example, suppose there are several di↵erent k -armed bandit tasks, and that oneach step you confront one of these chosen at random. Thus, the bandit task changes
randomly from step to step. If the probabilities with which each task is selected for you
do not change over time, this would appear as a single stationary k -armed bandit task,
and you could use one of the methods described in this chapter. Now suppose, however,that when a bandit task is selected for you, you are given some distinctive clue about its
identity (but not its action values). Maybe you are facing an actual slot machine thatchanges the color of its display as it changes its action values. Now you can learn a policy
associating each task, signaled by the color you see, with the best action to take whenfacing that task—for instance, if red, select arm 1; if green, select arm 2. With the rightpolicy you can usually do much better than you could in the absence of any information
distinguishing one bandit task from another.
This is an example of an associative search task, so called because it involves both
trial-and-error learning to search for the best actions, and association of these actions
with the situations in which they are best. Associative search tasks are often now called
contextual bandits in the literature. Associative search tasks are intermediate between
the k -armed bandit problem and the full reinforcement learning problem. They are likethe full reinforcement learning problem in that they involve learning a policy, but they
are also like our version of the k -armed bandit problem in that each action a↵ects only
the immediate reward. If actions are allowed to a↵ect the next situation as well as the
reward, then we have the full reinforcement learning problem. We present this problem
in the next chapter and consider its ramifications throughout the rest of the book.

Exercise 2.10 Suppose you face a 2-armed bandit task whose true action values change
randomly from time step to time step. Specifically, suppose that, for any time step,
the true values of actions 1 and 2 are respectively 10 and 20 with probability 0.5 (caseA), and 90 and 80 with probability 0.5 (case B). If you are not able to tell which caseyou face at any step, what is the best expected reward you can achieve and how should
you behave to achieve it? Now suppose that on each step you are told whether you arefacing case A or case B (although you still don’t know the true action values). This is anassociative search task. What is the best expected reward you can achieve in this task,
and how should you behave to achieve it?
⇤

42

2.10

Chapter 2: Multi-armed Bandits

Summary

We have presented in this chapter several simple ways of balancing exploration andexploitation. The "-greedy methods choose randomly a small fraction of the time, whereas
UCB methods choose deterministically but achieve exploration by subtly favoring at eachstep the actions that have so far received fewer samples. Gradient bandit algorithms
estimate not action values, but action preferences, and favor the more preferred actions
in a graded, probabilistic manner using a soft-max distribution. The simple expedient ofinitializing estimates optimistically causes even greedy methods to explore significantly.
It is natural to ask which of these methods is best. Although this is a difficult question
to answer in general, we can certainly run them all on the 10-armed testbed that wehave used throughout this chapter and compare their performances. A complication isthat they all have a parameter; to get a meaningful comparison we have to consider
their performance as a function of their parameter. Our graphs so far have shown thecourse of learning over time for each algorithm and parameter setting, to produce alearning curve for that algorithm and parameter setting. If we plotted learning curves
for all algorithms and all parameter settings, then the graph would be too complex andcrowded to make clear comparisons. Instead we summarize a complete learning curveby its average value over the 1000 steps; this value is proportional to the area under thelearning curve. Figure 2.6 shows this measure for the various bandit algorithms fromthis chapter, each as a function of its own parameter shown on a single scale on thex-axis. This kind of graph is called a parameter study. Note that the parameter values
are varied by factors of two and presented on a log scale. Note also the characteristic
inverted-U shapes of each algorithm’s performance; all the algorithms perform best at
an intermediate value of their parameter, neither too large nor too small. In assessing

1.5

greedy with
optimistic
initialization
α = 0.1

UCB
1.4

Averagerewardover first
1000 steps

1.3-greedy
gradient
bandit

1.2
1.1
1
1/128

1/64

1/32

1/16

1/8

1/4

1/2

1

2

4

" ↵ c Q0
Figure 2.6: A parameter study of the various bandit algorithms presented in this chapter.
Each point is the average reward obtained over 1000 steps with a particular algorithm at aparticular setting of its parameter.

2.10. Summary

43

a method, we should attend not just to how well it does at its best parameter setting,but also to how sensitive it is to its parameter value. All of these algorithms are fairly
insensitive, performing well over a range of parameter values varying by about an orderof magnitude. Overall, on this problem, UCB seems to perform best.
Despite their simplicity, in our opinion the methods presented in this chapter canfairly be considered the state of the art. There are more sophisticated methods, but theircomplexity and assumptions make them impractical for the full reinforcement learning
problem that is our real focus. Starting in Chapter 5 we present learning methods forsolving the full reinforcement learning problem that use in part the simple methods
explored in this chapter.
Although the simple methods explored in this chapter may be the best we can doat present, they are far from a fully satisfactory solution to the problem of balancing
exploration and exploitation.
One well-studied approach to balancing exploration and exploitation in k -armed bandit
problems is to compute a special kind of action value called a Gittins index. In certain
important special cases, this computation is tractable and leads directly to optimal
solutions, although it does require complete knowledge of the prior distribution of possible
problems, which we generally assume is not available. In addition, neither the theory
nor the computational tractability of this approach appear to generalize to the full
reinforcement learning problem that we consider in the rest of the book.
The Gittins-index approach is an instance of Bayesian methods, which assume a known
initial distribution over the action values and then update the distribution exactly after
each step (assuming that the true action values are stationary). In general, the update
computations can be very complex, but for certain special distributions (called conjugate
priors) they are easy. One possibility is to then select actions at each step according
to their posterior probability of being the best action. This method, sometimes called
posterior sampling or Thompson sampling, often performs similarly to the best of the
distribution-free methods we have presented in this chapter.
In the Bayesian setting it is even conceivable to compute the optimal balance between
exploration and exploitation. One can compute for any possible action the probabilityof each possible immediate reward and the resultant posterior distributions over action
values. This evolving distribution becomes the information state of the problem. Givena horizon, say of 1000 steps, one can consider all possible actions, all possible resulting
rewards, all possible next actions, all next rewards, and so on for all 1000 steps. Given
the assumptions, the rewards and probabilities of each possible chain of events can be
determined; one need only pick the best. But the tree of possibilities grows extremely
rapidly; even if there were only two actions and two rewards, the tree would have 22000
leaves. It is generally not feasible to perform this immense computation exactly, but
perhaps it could be approximated efficiently. This approach would e↵ectively turn thebandit problem into an instance of the full reinforcement learning problem. In the end, we
may be able to use approximate reinforcement learning methods such as those presented
in Part II of this book to approach this optimal solution. But that is a topic for researchand beyond the scope of this introductory book.

44

Chapter 2: Multi-armed Bandits

Exercise 2.11 (programming) Make a figure analogous to Figure 2.6 for the nonstationary
case outlined in Exercise 2.5. Include the constant-step-size "-greedy algorithm with
↵ = 0.1. Use runs of 200,000 steps and, as a performance measure for each algorithm andparameter setting, use the average reward over the last 100,000 steps.
⇤

Bibliographical and Historical Remarks
2.1

Bandit problems have been studied in statistics, engineering, and psychology. In
statistics, bandit problems fall under the heading “sequential design of experiments,” introduced by Thompson (1933, 1934) and Robbins (1952), and studied
by Bellman (1956). Berry and Fristedt (1985) provide an extensive treatment ofbandit problems from the perspective of statistics. Narendra and Thathachar(1989) treat bandit problems from the engineering perspective, providing a good
discussion of the various theoretical traditions that have focused on them. In
psychology, bandit problems have played roles in statistical learning theory (e.g.,
Bush and Mosteller, 1955; Estes, 1950).
The term greedy is often used in the heuristic search literature (e.g., Pearl, 1984).
The conflict between exploration and exploitation is known in control engineeringas the conflict between identification (or estimation) and control (e.g., Witten,
1976b). Feldbaum (1965) called it the dual control problem, referring to theneed to solve the two problems of identification and control simultaneously whentrying to control a system under uncertainty. In discussing aspects of genetic
algorithms, Holland (1975) emphasized the importance of this conflict, referring
to it as the conflict between the need to exploit and the need for new information.

2.2

Action-value methods for our k -armed bandit problem were first proposed byThathachar and Sastry (1985). These are often called estimator algorithms in thelearning automata literature. The term action value is due to Watkins (1989).
The first to use "-greedy methods may also have been Watkins (1989, p. 187),
but the idea is so simple that some earlier use seems likely.

2.4–5 This material falls under the general heading of stochastic iterative algorithms,
which is well covered by Bertsekas and Tsitsiklis (1996).
2.6

Optimistic initialization was used in reinforcement learning by Sutton (1996).

2.7

Early work on using estimates of the upper confidence bound to select actions
was done by Lai and Robbins (1985), Kaelbling (1993b), and Agrawal (1995).
The UCB algorithm we present here is called UCB1 in the literature and was
first developed by Auer, Cesa-Bianchi and Fischer (2002).

2.8

Gradient bandit algorithms are a special case of the gradient-based reinforcement
learning algorithms introduced by Williams (1992) that later developed into theactor–critic and policy-gradient algorithms that we treat later in this book. Ourdevelopment here was influenced by that by Balaraman Ravindran (personal

2.10. Summary

45

communication). Further discussion of the choice of baseline is provided by
Greensmith, Bartlett, and Baxter (2002, 2004) and by Dick (2015). Earlysystematic studies of algorithms like this were done by Sutton (1984).
The term soft-max for the action selection rule (2.11) is due to Bridle (1990).
This rule appears to have been first proposed by Luce (1959).
2.9

The term associative search and the corresponding problem were introduced by
Barto, Sutton, and Brouwer (1981). The term associative reinforcement learning
has also been used for associative search (Barto and Anandan, 1985), but we
prefer to reserve that term as a synonym for the full reinforcement learning
problem (as in Sutton, 1984). (And, as we noted, the modern literature alsouses the term “contextual bandits” for this problem.) We note that Thorndike’s
Law of E↵ect (quoted in Chapter 1) describes associative search by referringto the formation of associative links between situations (states) and actions.
According to the terminology of operant, or instrumental, conditioning (e.g.,
Skinner, 1938), a discriminative stimulus is a stimulus that signals the presenceof a particular reinforcement contingency. In our terms, di↵erent discriminative
stimuli correspond to di↵erent states.

2.10

Bellman (1956) was the first to show how dynamic programming could be used
to compute the optimal balance between exploration and exploitation within a
Bayesian formulation of the problem. The Gittins index approach is due to Gittins
and Jones (1974). Du↵ (1995) showed how it is possible to learn Gittins indicesfor bandit problems through reinforcement learning. The survey by Kumar (1985)
provides a good discussion of Bayesian and non-Bayesian approaches to these
problems. The term information state comes from the literature on partiallyobservable MDPs; see, for example, Lovejoy (1991).
Other theoretical research focuses on the efficiency of exploration, usually expressed as how quickly an algorithm can approach an optimal decision-making
policy. One way to formalize exploration efficiency is by adapting to reinforcement
learning the notion of sample complexity for a supervised learning algorithm,
which is the number of training examples the algorithm needs to attain a desireddegree of accuracy in learning the target function. A definition of the sample
complexity of exploration for a reinforcement learning algorithm is the number oftime steps in which the algorithm does not select near-optimal actions (Kakade,2003). Li (2012) discusses this and several other approaches in a survey of theoretical approaches to exploration efficiency in reinforcement learning. A thoroughmodern treatment of Thompson sampling is provided by Russo et al. (2018).

Chapter 3

Finite Markov Decision
Processes
In this chapter we introduce the formal problem of finite Markov decision processes, or
finite MDPs, which we try to solve in the rest of the book. This problem involves evaluative
feedback, as in bandits, but also an associative aspect—choosing di↵erent actions indi↵erent situations. MDPs are a classical formalization of sequential decision making,where actions influence not just immediate rewards, but also subsequent situations,or states, and through those future rewards. Thus MDPs involve delayed reward andthe need to trade o↵ immediate and delayed reward. Whereas in bandit problems weestimated the value q⇤ (a) of each action a, in MDPs we estimate the value q⇤ (s, a) ofeach action a in each state s, or we estimate the value v⇤ (s) of each state given optimal
action selections. These state-dependent quantities are essential to accurately assigningcredit for long-term consequences to individual action selections.
MDPs are a mathematically idealized form of the reinforcement learning problem
for which precise theoretical statements can be made. We introduce key elements ofthe problem’s mathematical structure, such as returns, value functions, and Bellman
equations. We try to convey the wide range of applications that can be formulated as
finite MDPs. As in all of artificial intelligence, there is a tension between breadth ofapplicability and mathematical tractability. In this chapter we introduce this tension
and discuss some of the trade-o↵s and challenges that it implies. Some ways in which
reinforcement learning can be taken beyond MDPs are treated in Chapter 17.

3.1

The Agent–Environment Interface

MDPs are meant to be a straightforward framing of the problem of learning frominteraction to achieve a goal. The learner and decision maker is called the agent. The
thing it interacts with, comprising everything outside the agent, is called the environment.
These interact continually, the agent selecting actions and the environment responding to

48

Chapter 3: Finite Markov Decision Processes

these actions and presenting new situations to the agent.1 The environment also gives
rise to rewards, special numerical values that the agent seeks to maximize over timethrough its choice of actions.

Agent
state

rewardSt

Rtaction
Rt+1
St+1

At

Environment

Figure 3.1: The agent–environment interaction in a Markov decision process.

More specifically, the agent and environment interact at each of a sequence of discrete
time steps, t = 0, 1, 2, 3, . . ..2 At each time step t, the agent receives some representationof the environment’s state, St 2 S, and on that basis selects an action, At 2 A(s).3 One
time step later, in part as a consequence of its action, the agent receives a numericalreward , Rt+1 2 R ⇢ R, and finds itself in a new state, St+1 .4 The MDP and agent
together thereby give rise to a sequence or trajectory that begins like this:S0 , A0 , R1 , S1 , A1 , R2 , S2 , A2 , R3 , . . .

(3.1)

In a finite MDP, the sets of states, actions, and rewards (S, A, and R) all have a finite
number of elements. In this case, the random variables Rt and St have well defined
discrete probability distributions dependent only on the preceding state and action. Thatis, for particular values of these random variables, s0 2 S and r 2 R, there is a probabilityof those values occurring at time t, given particular values of the preceding state andaction:.
p(s0 , r |s, a) = Pr{St = s0 , Rt = r | St 1 = s, At 1 = a},(3.2)

for all s0 , s 2 S, r 2 R, and a 2 A(s). The function p defines the dynamics of the MDP.
The dot over the equals sign in the equation reminds us that it is a definition (in thiscase of the function p) rather than a fact that follows from previous definitions. The
dynamics function p : S ⇥ R ⇥ S ⇥ A ! [0, 1] is an ordinary deterministic function of four
arguments. The ‘|’ in the middle of it comes from the notation for conditional probability,
1 We use the terms agent, environment, and action instead of the engineers’ terms controller, controlled
system (or plant), and control signal because they are meaningful to a wider audience.
2 We restrict attention to discrete time to keep things as simple as possible, even though many of theideas can be extended to the continuous-time case (e.g., see Bertsekas and Tsitsiklis, 1996; Doya, 1996).
3 To simplify notation, we sometimes assume the special case in which the action set is the same in allstates and write it simply as A.
4 We use R
t+1 instead of Rt to denote the reward due to At because it emphasizes that the next
reward and next state, Rt+1 and St+1 , are jointly determined. Unfortunately, both conventions arewidely used in the literature.

3.1. The Agent–Environment Interface

49

but here it just reminds us that p specifies a probability distribution for each choice of sand a, that is, thatXXp(s0 , r |s, a) = 1, for all s 2 S, a 2 A(s).
(3.3)
s0 2S r2R

In a Markov decision process, the probabilities given by p completely characterize theenvironment’s dynamics. That is, the probability of each possible value for St and Rt
depends on the immediately preceding state and action, St 1 and At 1 , and, given them,
not at all on earlier states and actions. This is best viewed as a restriction not on thedecision process, but on the state. The state must include information about all aspectsof the past agent–environment interaction that make a di↵erence for the future. If it
does, then the state is said to have the Markov property. We will assume the Markov
property throughout this book, though starting in Part II we will consider approximation
methods that do not rely on it, and in Chapter 17 we consider how a Markov state canbe efficiently learned and constructed from non-Markov observations.
From the four-argument dynamics function, p, one can compute anything else one mightwant to know about the environment, such as the state-transition probabilities (which wedenote, with a slight abuse of notation, as a three-argument function p : S⇥S⇥A ! [0, 1]),X.
p(s0 |s, a) = Pr{St = s0 | St 1 = s, At 1 = a} =p(s0 , r |s, a).

(3.4)

r2R

We can also compute the expected rewards for state–action pairs as a two-argument
function r : S ⇥ A ! R:
X X.
r(s, a) = E[Rt | St 1 = s, At 1 = a] =rp(s0 , r |s, a),
(3.5)
r2R

s0 2S

and the expected rewards for state–action–next-state triples as a three-argument functionr : S ⇥ A ⇥ S ! R,
X p(s0 , r |s, a).
r(s, a, s0 ) = E[Rt | St 1 = s, At 1 = a, St = s0 ] =r.
p(s0 |s, a)

(3.6)

r2R

In this book, we usually use the four-argument p function (3.2), but each of these othernotations are also occasionally convenient.
The MDP framework is abstract and flexible and can be applied to many di↵erent
problems in many di↵erent ways. For example, the time steps need not refer to fixedintervals of real time; they can refer to arbitrary successive stages of decision making
and acting. The actions can be low-level controls, such as the voltages applied to themotors of a robot arm, or high-level decisions, such as whether or not to have lunch or
to go to graduate school. Similarly, the states can take a wide variety of forms. They
can be completely determined by low-level sensations, such as direct sensor readings, orthey can be more high-level and abstract, such as symbolic descriptions of objects in aroom. Some of what makes up a state could be based on memory of past sensations or50

Chapter 3: Finite Markov Decision Processes

even be entirely mental or subjective. For example, an agent could be in the state of not
being sure where an object is, or of having just been surprised in some clearly defined
sense. Similarly, some actions might be totally mental or computational. For example,
some actions might control what an agent chooses to think about, or where it focuses its
attention. In general, actions can be any decisions we want to learn how to make, andstates can be anything we can know that might be useful in making them.
In particular, the boundary between agent and environment is typically not the sameas the physical boundary of a robot’s or an animal’s body. Usually, the boundary isdrawn closer to the agent than that. For example, the motors and mechanical linkages ofa robot and its sensing hardware should usually be considered parts of the environment
rather than parts of the agent. Similarly, if we apply the MDP framework to a person
or animal, the muscles, skeleton, and sensory organs should be considered part of theenvironment. Rewards, too, presumably are computed inside the physical bodies ofnatural and artificial learning systems, but are considered external to the agent.
The general rule we follow is that anything that cannot be changed arbitrarily by
the agent is considered to be outside of it and thus part of its environment. We do
not assume that everything in the environment is unknown to the agent. For example,
the agent often knows quite a bit about how its rewards are computed as a function of
its actions and the states in which they are taken. But we always consider the reward
computation to be external to the agent because it defines the task facing the agent andthus must be beyond its ability to change arbitrarily. In fact, in some cases the agent may
know everything about how its environment works and still face a difficult reinforcement
learning task, just as we may know exactly how a puzzle like Rubik’s cube works, but
still be unable to solve it. The agent–environment boundary represents the limit of theagent’s absolute control, not of its knowledge.
The agent–environment boundary can be located at di↵erent places for di↵erent
purposes. In a complicated robot, many di↵erent agents may be operating at once, eachwith its own boundary. For example, one agent may make high-level decisions which form
part of the states faced by a lower-level agent that implements the high-level decisions. In
practice, the agent–environment boundary is determined once one has selected particularstates, actions, and rewards, and thus has identified a specific decision-making task ofinterest.
The MDP framework is a considerable abstraction of the problem of goal-directed
learning from interaction. It proposes that whatever the details of the sensory, memory,and control apparatus, and whatever objective one is trying to achieve, any problem oflearning goal-directed behavior can be reduced to three signals passing back and forth
between an agent and its environment: one signal to represent the choices made by theagent (the actions), one signal to represent the basis on which the choices are made (thestates), and one signal to define the agent’s goal (the rewards). This framework may not
be sufficient to represent all decision-learning problems usefully, but it has proved to be
widely useful and applicable.
Of course, the particular states and actions vary greatly from task to task, and how
they are represented can strongly a↵ect performance. In reinforcement learning, as inother kinds of learning, such representational choices are at present more art than science.

3.1. The Agent–Environment Interface

51

In this book we o↵er some advice and examples regarding good ways of representing
states and actions, but our primary focus is on general principles for learning how tobehave once the representations have been selected.
Example 3.1: Bioreactor Suppose reinforcement learning is being applied to determine
moment-by-moment temperatures and stirring rates for a bioreactor (a large vat ofnutrients and bacteria used to produce useful chemicals). The actions in such an
application might be target temperatures and target stirring rates that are passed tolower-level control systems that, in turn, directly activate heating elements and motors toattain the targets. The states are likely to be thermocouple and other sensory readings,
perhaps filtered and delayed, plus symbolic inputs representing the ingredients in thevat and the target chemical. The rewards might be moment-by-moment measures of therate at which the useful chemical is produced by the bioreactor. Notice that here eachstate is a list, or vector, of sensor readings and symbolic inputs, and each action is avector consisting of a target temperature and a stirring rate. It is typical of reinforcement
learning tasks to have states and actions with such structured representations. Rewards,
on the other hand, are always single numbers.
Example 3.2: Pick-and-Place Robot Consider using reinforcement learning tocontrol the motion of a robot arm in a repetitive pick-and-place task. If we want to learn
movements that are fast and smooth, the learning agent will have to control the motors
directly and have low-latency information about the current positions and velocitiesof the mechanical linkages. The actions in this case might be the voltages applied toeach motor at each joint, and the states might be the latest readings of joint angles andvelocities. The reward might be +1 for each object successfully picked up and placed. To
encourage smooth movements, on each time step a small, negative reward could be given
as a function of the moment-to-moment jerkiness of the motion.
Exercise 3.1 Devise three example tasks of your own that fit into the MDP framework,
identifying for each its states, actions, and rewards. Make the three examples as di↵erent
from each other as possible. The framework is abstract and flexible and can be applied inmany di↵erent ways. Stretch its limits in some way in at least one of your examples. ⇤Exercise 3.2 Is the MDP framework adequate to usefully represent all goal-directed
learning tasks? Can you think of any clear exceptions?
⇤
Exercise 3.3 Consider the problem of driving. You could define the actions in terms of
the accelerator, steering wheel, and brake, that is, where your body meets the machine.
Or you could define them farther out—say, where the rubber meets the road, considering
your actions to be tire torques. Or you could define them farther in—say, where yourbrain meets your body, the actions being muscle twitches to control your limbs. Or you
could go to a really high level and say that your actions are your choices of where to drive.
What is the right level, the right place to draw the line between agent and environment?
On what basis is one location of the line to be preferred over another? Is there any
fundamental reason for preferring one location over another, or is it a free choice?
⇤

52

Chapter 3: Finite Markov Decision Processes

Example 3.3

Recycling Robot

A mobile robot has the job of collecting empty soda cans in an office environment. It
has sensors for detecting cans, and an arm and gripper that can pick them up and placethem in an onboard bin; it runs on a rechargeable battery. The robot’s control system
has components for interpreting sensory information, for navigating, and for controllingthe arm and gripper. High-level decisions about how to search for cans are made by areinforcement learning agent based on the current charge level of the battery. To make asimple example, we assume that only two charge levels can be distinguished, comprisinga small state set S = {high, low}. In each state, the agent can decide whether to (1)actively search for a can for a certain period of time, (2) remain stationary and wait
for someone to bring it a can, or (3) head back to its home base to recharge its battery.
When the energy level is high, recharging would always be foolish, so we do not include itin the action set for this state. The action sets are then A(high) = {search, wait} andA(low) = {search, wait, recharge}.

The rewards are zero most of the time, but become positive when the robot secures anempty can, or large and negative if the battery runs all the way down. The best way tofind cans is to actively search for them, but this runs down the robot’s battery, whereas
waiting does not. Whenever the robot is searching, the possibility exists that its battery
will become depleted. In this case the robot must shut down and wait to be rescued(producing a low reward). If the energy level is high, then a period of active search canalways be completed without risk of depleting the battery. A period of searching thatbegins with a high energy level leaves the energy level high with probability ↵ and reduces
it to low with probability 1 ↵. On the other hand, a period of searching undertaken
when the energy level is low leaves it low with probability and depletes the battery withprobability 1. In the latter case, the robot must be rescued, and the battery is thenrecharged back to high. Each can collected by the robot counts as a unit reward, whereasa reward of 3 results whenever the robot has to be rescued. Let rsearch and rwait , withrsearch > rwait , denote the expected number of cans the robot will collect (and hence theexpected reward) while searching and while waiting respectively. Finally, suppose that nocans can be collected during a run home for recharging, and that no cans can be collected
on a step in which the battery is depleted. This system is then a finite MDP, and we
can write down the transition probabilities and the expected rewards, with dynamics asindicated in the table on the left:
s
high
high
low
low
high
high
low
low
low
low

a
searchsearchsearchsearch
wait
wait
wait
wait
recharge
recharge

s0
high
low
high
low
high
low
high
low
high
low

p(s0 |s, a)
↵
1 ↵
1
1
0
0
1
1
0

r(s, a, s0 )rsearchrsearch3
rsearch
rwait
rwait
0
-

1, rwait1, rsearch, 3
search

wait

high

1, 0

recharge

wait

search↵, rsearchlow

1 ↵, rsearch

1, rwait

Note that there is a row in the table for each possible combination of current state, s,
action, a 2 A(s), and next state, s0 . Some transitions have zero probability of occurring,so no expected reward is specified for them. Shown on the right is another useful way of3.2. Goals and Rewards

53

summarizing the dynamics of a finite MDP, as a transition graph. There are two kinds ofnodes: state nodes and action nodes. There is a state node for each possible state (a large
open circle labeled by the name of the state), and an action node for each state–action
pair (a small solid circle labeled by the name of the action and connected by a line to thestate node). Starting in state s and taking action a moves you along the line from statenode s to action node (s, a). Then the environment responds with a transition to the next
state’s node via one of the arrows leaving action node (s, a). Each arrow corresponds toa triple (s, s0 , a), where s0 is the next state, and we label the arrow with the transition
probability, p(s0 |s, a), and the expected reward for that transition, r(s, a, s0 ). Note thatthe transition probabilities labeling the arrows leaving an action node always sum to 1.

Exercise 3.4 Give a table analogous to that in Example 3.3, but for p(s0 , r |s, a). It
should have columns for s, a, s0 , r, and p(s0 , r |s, a), and a row for every 4-tuple for whichp(s0 , r |s, a) > 0.
⇤

3.2

Goals and Rewards

In reinforcement learning, the purpose or goal of the agent is formalized in terms of a
special signal, called the reward, passing from the environment to the agent. At each time
step, the reward is a simple number, Rt 2 R. Informally, the agent’s goal is to maximize
the total amount of reward it receives. This means maximizing not immediate reward,
but cumulative reward in the long run. We can clearly state this informal idea as the
reward hypothesis:
That all of what we mean by goals and purposes can be well thought of as
the maximization of the expected value of the cumulative sum of a received
scalar signal (called reward).
The use of a reward signal to formalize the idea of a goal is one of the most distinctive
features of reinforcement learning.
Although formulating goals in terms of reward signals might at first appear limiting,in practice it has proved to be flexible and widely applicable. The best way to see this is
to consider examples of how it has been, or could be, used. For example, to make a robot
learn to walk, researchers have provided reward on each time step proportional to therobot’s forward motion. In making a robot learn how to escape from a maze, the reward
is often 1 for every time step that passes prior to escape; this encourages the agent toescape as quickly as possible. To make a robot learn to find and collect empty soda cansfor recycling, one might give it a reward of zero most of the time, and then a reward of+1 for each can collected. One might also want to give the robot negative rewards when
it bumps into things or when somebody yells at it. For an agent to learn to play checkers
or chess, the natural rewards are +1 for winning, 1 for losing, and 0 for drawing andfor all nonterminal positions.
You can see what is happening in all of these examples. The agent always learns tomaximize its reward. If we want it to do something for us, we must provide rewards
to it in such a way that in maximizing them the agent will also achieve our goals. It

54

Chapter 3: Finite Markov Decision Processes

is thus critical that the rewards we set up truly indicate what we want accomplished.
In particular, the reward signal is not the place to impart to the agent prior knowledge
about how to achieve what we want it to do.5 For example, a chess-playing agent should
be rewarded only for actually winning, not for achieving subgoals such as taking its
opponent’s pieces or aining control of the center of the board. If achieving these sortsof subgoals were rewarded, then the agent might find a way to achieve them withoutachieving the real goal. For example, it might find a way to take the opponent’s pieces
even at the cost of losing the game. The reward signal is your way of communicating to
the agent what you want achieved, not how you want it achieved.6

3.3

Returns and Episodes

So far we have discussed informally the objective of learning. We have said that theagent’s goal is to maximize the cumulative reward it receives in the long run. How might
this be defined formally? If the sequence of rewards received after time step t is denoted
Rt+1 , Rt+2 , Rt+3 , . . ., then what precise aspect of this sequence do we wish to maximize?
In general, we seek to maximize the expected return, where the return, denoted Gt , isdefined as some specific function of the reward sequence. In the simplest case the returnis the sum of the rewards:.
Gt = Rt+1 + Rt+2 + Rt+3 + · · · + RT ,(3.7)

where T is a final time step. This approach makes sense in applications in which there
is a natural notion of final time step, that is, when the agent–environment interaction
breaks naturally into subsequences, which we call episodes,7 such as plays of a game,trips through a maze, or any sort of repeated interaction. Each episode ends in a special
state called the terminal state, followed by a reset to a standard starting state or to asample from a standard distribution of starting states. Even if you think of episodes as
ending in di↵erent ways, such as winning and losing a game, the next episode begins
independently of how the previous one ended. Thus the episodes can all be considered to
end in the same terminal state, with di↵erent rewards for the di↵erent outcomes. Tasks
with episodes of this kind are called episodic tasks. In episodic tasks we sometimes needto distinguish the set of all nonterminal states, denoted S, from the set of all states plus
the terminal state, denoted S+ . The time of termination, T , is a random variable that
normally varies from episode to episode.
On the other hand, in many cases the agent–environment interaction does not breaknaturally into identifiable episodes, but goes on continually without limit. For example,
this would be the natural way to formulate an on-going process-control task, or an
application to a robot with a long life span. We call these continuing tasks. The return
formulation (3.7) is problematic for continuing tasks because the final time step would be
T = 1, and the return, which is what we are trying to maximize, could easily be infinite.
5 Better places for imparting this kind of prior knowledge are the initial policy or initial value function.
6 Section 17.4 delves further into the issue of designing e↵ective reward signals.
7 Episodes are sometimes called “trials” in the literature.

3.3. Returns and Episodes

55(For example, suppose the agent receives a reward of +1 at each time step.) Thus, in thisbook we usually use a definition of return that is slightly more complex conceptually but
much simpler mathematically.
The additional concept that we need is that of discounting. According to this approach,the agent tries to select actions so that the sum of the discounted rewards it receives overthe future is maximized. In particular, it chooses At to maximize the expected discounted
return:.
Gt = Rt+1 + Rt+2 +2

Rt+3 + · · · =1Xk

Rt+k+1 ,

(3.8)

k=0

where is a parameter, 0   1, called the discount rate.
The discount rate determines the present value of future rewards: a reward received
k time steps in the future is worth only k 1 times what it would be worth if it werereceived immediately. If < 1, the infinite sum in (3.8) has a finite value as long as the
reward sequence {Rk } is bounded. If = 0, the agent is “myopic” in being concerned
only with maximizing immediate rewards: its objective in this case is to learn how tochoose At so as to maximize only Rt+1 . If each of the agent’s actions happened toinfluence only the immediate reward, not future rewards as well, then a myopic agent
could maximize (3.8) by separately maximizing each immediate reward. But in general,
acting to maximize immediate reward can reduce access to future rewards so that thereturn is reduced. As approaches 1, the return objective takes future rewards into
account more strongly; the agent becomes more farsighted.
Returns at successive time steps are related to each other in a way that is important
for the theory and algorithms of reinforcement learning:.
Gt = Rt+1 + Rt+2 += Rt+1 +Rt+3 +

3

Rt+2 + Rt+3 +2= Rt+1 + Gt+1

2

Rt+4 + · · ·Rt+4 + · · ·(3.9)

Note that this works for all time steps t < T , even if termination occurs at t + 1, provided
we define GT = 0. This often makes it easy to compute returns from reward sequences.
Note that although the return (3.8) is a sum of an infinite number of terms, it is still
finite if the reward is nonzero and constant—if < 1. For example, if the reward is aconstant +1, then the return is
Gt =

1
X

k=0

k

=

1
1

.

(3.10)

Exercise 3.5 The equations in Section 3.1 are for the continuing case and need to bemodified (very slightly) to apply to episodic tasks. Show that you know the modifications
needed by giving the modified version of (3.3).
⇤

56

Chapter 3: Finite Markov Decision Processes

Example 3.4: Pole-Balancing
The objective in this task is to apply
forces to a cart moving along a trackso as to keep a pole hinged to the cartfrom falling over: A failure is said tooccur if the pole falls past a given angle
from vertical or if the cart runs o↵ thetrack. The pole is reset to verticalafter each failure. This task could betreated as episodic, where the naturalepisodes are the repeated attempts to balance the pole. The reward in this case could be+1 for every time step on which failure did not occur, so that the return at each time
would be the number of steps until failure. In this case, successful balancing forever would
mean a return of infinity. Alternatively, we could treat pole-balancing as a continuingtask, using discounting. In this case the reward would be 1 on each failure and zero at
K 1
all other times. The return at each time would then be related to, where K is
the number of time steps before failure (as well as to the times of later failures). In either
case, the return is maximized by keeping the pole balanced for as long as possible.
Exercise 3.6 Suppose you treated pole-balancing as an episodic task but also useddiscounting, with all rewards zero except for 1 upon failure. What then would thereturn be at each time? How does this return di↵er from that in the discounted, continuing
formulation of this task?
⇤
Exercise 3.7 Imagine that you are designing a robot to run a maze. You decide to give it areward of +1 for escaping from the maze and a reward of zero at all other times. The task
seems to break down naturally into episodes—the successive runs through the maze—so
you decide to treat it as an episodic task, where the goal is to maximize expected total
reward (3.7). After running the learning agent for a while, you find that it is showing
no improvement in escaping from the maze. What is going wrong? Have you e↵ectively
communicated to the agent what you want it to achieve?
⇤
Exercise 3.8 Suppose = 0.5 and the following sequence of rewards is received R1 = 1,R2 = 2, R3 = 6, R4 = 3, and R5 = 2, with T = 5. What are G0 , G1 , . . ., G5 ? Hint:
Work backwards.
⇤
Exercise 3.9 Suppose = 0.9 and the reward sequence is R1 = 2 followed by an infinite
sequence of 7s. What are G1 and G0 ?
⇤Exercise 3.10 Prove the second equality in (3.10).

⇤

3.4. Unified Notation for Episodic and Continuing Tasks

3.4

57

Unified Notation for Episodic and Continuing Tasks

In the preceding section we described two kinds of reinforcement learning tasks, one
in which the agent–environment interaction naturally breaks down into a sequence ofseparate episodes (episodic tasks), and one in which it does not (continuing tasks). The
former case is mathematically easier because each action a↵ects only the finite number ofrewards subsequently received during the episode. In this book we consider sometimes
one kind of problem and sometimes the other, but often both. It is therefore useful toestablish one notation that enables us to talk precisely about both cases simultaneously.
To be precise about episodic tasks requires some additional notation. Rather than one
long sequence of time steps, we need to consider a series of episodes, each of which consistsof a finite sequence of time steps. We number the time steps of each episode starting
anew from zero. Therefore, we have to refer not just to St , the state representation attime t, but to St,i , the state representation at time t of episode i (and similarly for At,i ,Rt,i , ⇡t,i , Ti , etc.). However, it turns out that when we discuss episodic tasks we almost
never have to distinguish between di↵erent episodes. We are almost always consideringa particular episode, or stating something that is true for all episodes. Accordingly, inpractice we almost always abuse notation slightly by dropping the explicit reference toepisode number. That is, we write St to refer to St,i , and so on.
We need one other convention to obtain a single notation that covers both episodic
and continuing tasks. We have defined the return as a sum over a finite number of terms
in one case (3.7) and as a sum over an infinite number of terms in the other (3.8). These
two can be unified by considering episode termination to be the entering of a special
absorbing state that transitions only to itself and that generates only rewards of zero. For
example, consider the state transition diagram:

S0

R1 = +1

S1

R2 = +1

S2

R3 = +1

R4 = 0R5 = 0...
Here the solid square represents the special absorbing state corresponding to the end of an
episode. Starting from S0 , we get the reward sequence +1, +1, +1, 0, 0, 0, . . .. Summingthese, we get the same return whether we sum over the first T rewards (here T = 3) orover the full infinite sequence. This remains true even if we introduce discounting. Thus,
we can define the return, in general, according to (3.8), using the convention of omitting
episode numbers when they are not needed, and including the possibility that = 1 if thesum remains defined (e.g., because all episodes terminate). Alternatively, we can write
T
. X
Gt =

k t 1

Rk ,

(3.11)

k=t+1

including the possibility that T = 1 or = 1 (but not both). We use these conventions
throughout the rest of the book to simplify notation and to express the close parallels

58

Chapter 3: Finite Markov Decision Processes

between episodic and continuing tasks. (Later, in Chapter 10, we will introduce aformulation that is both continuing and undiscounted.)

3.5

Policies and Value Functions

Almost all reinforcement learning algorithms involve estimating value functions—functionsof states (or of state–action pairs) that estimate how good it is for the agent to be in agiven state (or how good it is to perform a given action in a given state). The notionof “how good” here is defined in terms of future rewards that can be expected, or, tobe precise, in terms of expected return. Of course the rewards the agent can expect toreceive in the future depend on what actions it will take. Accordingly, value functions
are defined with respect to particular ways of acting, called policies.
Formally, a policy is a mapping from states to probabilities of selecting each possible
action. If the agent is following policy ⇡ at time t, then ⇡(a|s) is the probability thatAt = a if St = s. Like p, ⇡ is an ordinary function; the “|” in the middle of ⇡(a|s)
merely reminds us that it defines a probability distribution over a 2 A(s) for each s 2 S.
Reinforcement learning methods specify how the agent’s policy is changed as a result of
its experience.
Exercise 3.11 If the current state is St , and actions are selected according to a stochastic
policy ⇡, then what is the expectation of Rt+1 in terms of ⇡ and the four-argument
function p (3.2)?
⇤
The value function of a state s under a policy ⇡, denoted v⇡ (s), is the expected return
when starting in s and following ⇡ thereafter. For MDPs, we can define v⇡ formally by
"1
#
X
.
k
v⇡ (s) = E⇡[Gt | St = s] = E⇡
Rt+k+1 St = s , for all s 2 S,
(3.12)
k=0

where E⇡[·] denotes the expected value of a random variable given that the agent follows
policy ⇡, and t is any time step. Note that the value of the terminal state, if any, isalways zero. We call the function v⇡ the state-value function for policy ⇡.
Similarly, we define the value of taking action a in state s under a policy ⇡, denotedq⇡ (s, a), as the expected return starting from s, taking the action a, and thereafter
following policy ⇡:
"1
#
X
.
k
q⇡ (s, a) = E⇡[Gt | St = s, At = a] = E⇡
Rt+k+1 St = s, At = a . (3.13)
k=0

We call q⇡ the action-value function for policy ⇡.
Exercise 3.12 Give an equation for v⇡ in terms of q⇡ and ⇡.

⇤Exercise 3.13 Give an equation for q⇡ in terms of v⇡ and the four-argument p.
⇤
The value functions v⇡ and q⇡ can be estimated from experience. For example, if anagent follows policy ⇡ and maintains an average, for each state encountered, of the actual
returns that have followed that state, then the average will converge to the state’s value,v⇡ (s), as the number of times that state is encountered approaches infinity. If separate

3.5. Policies and Value Functions

59

averages are kept for each action taken in each state, then these averages will similarly
converge to the action values, q⇡ (s, a). We call estimation methods of this kind Monte
Carlo methods because they involve averaging over many random samples of actual returns.
These kinds of methods are presented in Chapter 5. Of course, if there are very manystates, then it may not be practical to keep separate averages for each state individually.
Instead, the agent would have to maintain v⇡ and q⇡ as parameterized functions (withfewer parameters than states) and adjust the parameters to better match the observed
returns. This can also produce accurate estimates, although much depends on the natureof the parameterized function approximator. These possibilities are discussed in Part IIof the book.
A fundamental property of value functions used throughout reinforcement learning anddynamic programming is that they satisfy recursive relationships similar to that which
we have already established for the return (3.9). For any policy ⇡ and any state s, thefollowing consistency condition holds between the value of s and the value of its possible
successor states:.
v⇡ (s) = E⇡[Gt | St = s]
= E⇡[Rt+1 + Gt+1 | St = s]
h
i
X
XX
=
⇡(a|s)p(s0 , r |s, a) r + E⇡[Gt+1 |St+1 = s0 ](by (3.9))

for all s 2 S,

(3.14)

s0

a

=

X
a

⇡(a|s)

X
s0 ,r

r

h
i
p(s0 , r |s, a) r + v⇡ (s0 ) ,where it is implicit that the actions, a, are taken from the set A(s), that the next states,
s0 , are taken from the set S (or from S+ in the case of an episodic problem), and thatthe rewards, r, are taken from the set R. Note also how in the last equation we havemerged the two sums, one over all the values of s0 and the other over all the values of r,
into one sum over all the possible values of both. We use this kind of merged sum often
to simplify formulas. Note how the final expression can be read easily as an expectedvalue. It is really a sum over all values of the three variables, a, s0 , and r. For each triple,
we compute its probability, ⇡(a|s)p(s0 , r |s, a), weight the quantity in brackets by thatprobability, then sum over all possibilities to get an expected value.
Equation (3.14) is the Bellman equation for v⇡ . It expressessa relationship between the value of a state and the values of
its successor states. Think of looking ahead from a state to its⇡apossible successor states, as suggested by the diagram to the⇡right. Each open circle represents a state and each solid circle
p r
represents a state–action pair. Starting from state s, the roots0
node at the top, the agent could take any of some set of actions—
Backup diagram for v⇡
three are shown in the diagram—based on its policy ⇡. Fromeach of these, the environment could respond with one of several next states, s0 (two areshown in the figure), along with a reward, r, depending on its dynamics given by thefunction p. The Bellman equation (3.14) averages over all the possibilities, weighting eachby its probability of occurring. It states that the value of the start state must equal the(discounted) value of the expected next state, plus the reward expected along the way.

the states of the environment. At each cell, four actions are possible: north,south, east, and west, which deterministically cause the agent to move one
cell in the respective direction on the grid. Actions that would take the agent
off thebut also
resultMarkov
in a Decision
reward Processesof 1.
60 grid leave its location unchanged, Chapter
3: Finite
Other actions result in a reward of 0, except those that move the agent outof the special states A and B. From state A, all four actions yield a reward of
The value function v⇡ is the unique solution to its Bellman equation. We show in+10subsequentand take chapters
the agent
A Bellman. From equation
state B,forms
all actionsa rewardof +5
howtothisthe basisyieldof a numberof waystoand compute,take theapproximate,agent to Band. learn v⇡ . We call diagrams like that above backup diagrams
because they diagram relationships that form the basis of the update or backup operations

Supposethe agent selects all four actions with equal probability in allthat are at the heart of reinforcement learning methods. These operations transfer
states.
3.5b back
shows
value
v⇡ pair), for fromthis its
policy,for states
the disvalueFigure
information
to athestate(or afunction,state–action
successor(orcountedrewardpairs).
case with= 0.9. diagrams
This valuefunctioncomputedsolvstate–action
We use backup
throughout
thewasbook
to provide bygraphical
summaries
theequations
algorithms (3.10).
we discuss.
(Notethethat,
unlike transition
graphs,state
ing thesystemof of
Noticenegativevalues nearthethelower
nodes of backup diagrams do not necessarily represent distinct states; for example, aedge; these are the result of the high probability of hitting the edge of the gridstate might be its own successor.)
there under the random policy. State A is the best state to be in under this polExample
3.5: Gridworld
3.2 (left)a rectangularreward,
gridworld
representationicy, butits expected
return isFigure
less than
10,shows
its immediate
becausefromof a simple finite MDP. The cells of the grid correspond to the states of the environment. At
A the
agent
is taken, from north,whichsouth,
it is likely
to runtheedge of theeachcell, four
actionsto
areApossible:
east, andwest,into
which
deterministicallygrid.causeStateon theotheronehand,
is thevaluedmore direction
than 5, its
immediatereward,theB,agent
to movecell inrespectiveon thegrid. Actions
thatwouldfromtake Bthetheagento↵ thegrid leave
unchanged,
but value.
also result
in aB
rewardbecauseagent
is taken
to Bits, location
which hasa positive
From
the
of
1.
Other
actions
resultinarewardof0,
except
those
that
move
the
agent
out
of
the
expected penalty (negative reward) for possibly running into an edge is more
special states A and B. From state A, all four actions yield a reward of +10 and take theagent to A0 . From state B, all actions yield a reward of +5 and take the agent to B0 .
A

+10

B

3.3 8.8 4.4 5.3 1.5+5

1.5 3.0 2.3 1.9 0.5

B'

0.1 0.7 0.7 0.4 -0.4
-1.0 -0.4 -0.4 -0.6 -1.2

ActionsA'(a)-1.9 -1.3 -1.2 -1.4 -2.0(b)Figure 3.2: Gridworld example: exceptional reward dynamics (left) and state-value function
for the equiprobable random policy (right).

Figure 3.5: Grid example: (a) exceptional reward dynamics; (b) state-valuefunctionfor therandompolicy.
Supposethe equiprobableagent selects all four
actionswith equal probability in all states. Figure 3.2(right) shows the value function, v⇡ , for this policy, for the discounted reward case with
= 0.9. This value function was computed by solving the system of linear equations(3.14). Notice the negative values near the lower edge; these are the result of the highprobability of hitting the edge of the grid there under the random policy. State A isthe best state to be in under this policy. Note that A’s expected return is less than its
immediate reward of 10, because from A the agent is taken to state A0 from which it islikely to run into the edge of the grid. State B, on the other hand, is valued more thanits immediate reward of 5, because from B the agent is taken to B0 which has a positive
value. From B0 the expected penalty (negative reward) for possibly running into an edge
is more than compensated for by the expected gain for possibly stumbling onto A or B.
Exercise 3.14 The Bellman equation (3.14) must hold for each state for the value functionv⇡ shown in Figure 3.2 (right) of Example 3.5. Show numerically that this equation holds
for the center state, valued at +0.7, with respect to its four neighboring states, valued at+2.3, +0.4, 0.4, and +0.7. (These numbers are accurate only to one decimal place.) ⇤3.5. Policies and Value Functions

61

Exercise 3.15 In the gridworld example, rewards are positive for goals, negative forrunning into the edge of the world, and zero the rest of the time. Are the signs of theserewards important, or only the intervals between them? Prove, using (3.8), that adding a
constant c to all the rewards adds a constant, vc , to the values of all states, and thus
does not a↵ect the relative values of any states under any policies. What is vc in termsof c and ?
⇤Exercise 3.16 Now consider adding a constant c to all the rewards in an episodic task,such as maze running. Would this have any e↵ect, or would it leave the task unchanged
as in the continuing task above? Why or why not? Give an example.
⇤
Example 3.6: Golf To formulate playing a hole of golf as a reinforcement learningtask, we count a penalty (negative reward) of 1 for each stroke until we hit the ballinto the hole. The state is the location of the ball. The value of a state is the negative of
the number of strokes to the hole from that location. Our actions are how we aim andswing at the ball, of course, and which club we select. Let us take the former as given
and consider just the choice of club, which we assume is either a putter or a driver. The
upper part of Figure 3.3 shows a possible state-value function, vputt (s), for the policy thatalways uses the putter. The terminal!4
state in-the-hole has a value of 0. From vputt!3
V puttanywhere on the green we assume we cansand!"make a putt; these states have value 1.
!1
O↵ the green we cannot reach the hole by
!2
!2
!3
!1
putting, and the value is lower. If we can0!4sreach the green from a state by putting,
green
a
!5
n
!6
d
then that state must have value one less!4 !"than the green’s value, that is, 2. For!2!3
simplicity, let us assume we can putt very
precisely and deterministically, but with*(s,driver)driver)a limited range. This gives us the sharp qQ
⇤ (s,
sand
contour line labeled 2 in the figure; alllocations between that line and the green
require exactly two strokes to complete
0 !1
!2
!2
the hole. Similarly, any location within
s
a
green
!3
n
putting range of the 2 contour line mustdhave a value of 3, and so on to get all thecontour lines shown in the figure. Putting
doesn’t get us out of sand traps, so they
Figure 3.3: A golf example: the state-value funchave a value of 1. Overall, it takes ustion for putting (upper) and the optimal actionsix strokes to get from the tee to the hole value function for using the driver (lower).
by putting.
s, aExercise 3.17 What is the Bellman equation for action values, thatp r
is, for q⇡ ? It must give the action value q⇡ (s, a) in terms of the actions0
0 0
values, q (s , a ), of possible successors to the state–action pair (s, a).
⇡

Hint: The backup diagram to the right corresponds to this equation.
Show the sequence of equations analogous to (3.14), but for action
values.
⇤⇡a0q⇡ backup diagram

62

Chapter 3: Finite Markov Decision Processes

Exercise 3.18 The value of a state depends on the values of the actions possible in thatstate and on how likely each action is to be taken under the current policy. We canthink of this in terms of a small backup diagram rooted at the state and considering eachpossible action:sv⇡ (s)
taken withprobability ⇡(a|s)

q⇡ (s, a)
a1

a2

a3

Give the equation corresponding to this intuition and diagram for the value at the root
node, v⇡ (s), in terms of the value at the expected leaf node, q⇡ (s, a), given St = s. Thisequation should include an expectation conditioned on following the policy, ⇡. Then givea second equation in which the expected value is written out explicitly in terms of ⇡(a|s)
such that no expected value notation appears in the equation.
⇤
Exercise 3.19 The value of an action, q⇡ (s, a), depends on the expected next reward andthe expected sum of the remaining rewards. Again we can think of this in terms of asmall backup diagram, this one rooted at an action (state–action pair) and branching to
the possible next states:s, aexpectedq⇡ (s, a)rewards

r1
s01

r2
s02

r3

v⇡ (s0 )s03

Give the equation corresponding to this intuition and diagram for the action value,q⇡ (s, a), in terms of the expected next reward, Rt+1 , and the expected next state value,v⇡ (St+1 ), given that St = s and At = a. This equation should include an expectation but
not one conditioned on following the policy. Then give a second equation, writing out the
expected value explicitly in terms of p(s0 , r |s, a) defined by (3.2), such that no expectedvalue notation appears in the equation.
⇤

3.6

Optimal Policies and Optimal Value Functions

Solving a reinforcement learning task means, roughly, finding a policy that achieves a lotof reward over the long run. For finite MDPs, we can precisely define an optimal policyin the following way. Value functions define a partial ordering over policies. A policy ⇡ isdefined to be better than or equal to a policy ⇡ 0 if its expected return is greater thanor equal to that of ⇡ 0 for all states. In other words, ⇡ ⇡ 0 if and only if v⇡ (s) v⇡0 (s)for all s 2 S. There is always at least one policy that is better than or equal to all otherpolicies. This is an optimal policy. Although there may be more than one, we denote all
the optimal policies by ⇡⇤ . They share the same state-value function, called the optimal
state-value function, denoted v⇤ , and defined as.
v⇤ (s) = max v⇡ (s),⇡for all s 2 S.

(3.15)

3.6. Optimal Policies and Optimal Value Functions

63

Optimal policies also share the same optimal action-value function, denoted q⇤ , anddefined as.
q⇤ (s, a) = max q⇡ (s, a),⇡(3.16)

for all s 2 S and a 2 A(s). For the state–action pair (s, a), this function gives the
expected return for taking action a in state s and thereafter following an optimal policy.
Thus, we can write q⇤ in terms of v⇤ as follows:q⇤ (s, a) = E[Rt+1 + v⇤ (St+1 ) | St = s, At = a] .

(3.17)

Example 3.7: Optimal Value Functions for Golf The lower part of Figure 3.3
shows the contours of a possible optimal action-value function q⇤ (s, driver). These are
the values of each state if we first play a stroke with the driver and afterward select either
the driver or the putter, whichever is better. The driver enables us to hit the ball farther,
but with less accuracy. We can reach the hole in one shot using the driver only if weare already very close; thus the 1 contour for q⇤ (s, driver) covers only a small portionof the green. If we have two strokes, however, then we can reach the hole from much
farther away, as shown by the 2 contour. In this case we don’t have to drive all the way
to within the small 1 contour, but only to anywhere on the green; from there we canuse the putter. The optimal action-value function gives the values after committing to aparticular first action, in this case, to the driver, but afterward using whichever actions
are best. The 3 contour is still farther out and includes the starting tee. From the tee,the best sequence of actions is two drives and one putt, sinking the ball in three strokes.
Because v⇤ is the value function for a policy, it must satisfy the self-consistency
condition given by the Bellman equation for state values (3.14). Because it is the optimalvalue function, however, v⇤ ’s consistency condition can be written in a special form
without reference to any specific policy. This is the Bellman equation for v⇤ , or the
Bellman optimality equation. Intuitively, the Bellman optimality equation expresses thefact that the value of a state under an optimal policy must equal the expected return forthe best action from that state:v⇤ (s) = max q⇡⇤ (s, a)a2A(s)= max E⇡⇤[Gt | St = s, At = a]a= max E⇡⇤[Rt+1 + Gt+1 | St = s, At = a]a= max E[Rt+1 + v⇤ (St+1 ) | St = s, At = a]
a
X
⇥
⇤
= max
p(s0 , r |s, a) r + v⇤ (s0 ) .
a

s0 ,r

(by (3.9))(3.18)(3.19)

64

Chapter 3: Finite Markov Decision Processes

The last two equations are two forms of the Bellman optimality equation for v⇤ . The
Bellman optimality equation for q⇤ ishi0q⇤ (s, a) = E Rt+1 + max
q
(S
,
a
)
S
=
s,
A
=
a
⇤
t+1
t
t
a0
h
i
X
0
0 0
=
p(s , r |s, a) r + max
q
(s
,
a
)
.
(3.20)
⇤
0
a

s0 ,r

The backup diagrams in the figure below show graphically the spans of future states
and actions considered in the Bellman optimality equations for v⇤ and q⇤ . These are thesame as the backup diagrams for v⇡ and q⇡ presented earlier except that arcs have been
added at the agent’s choice points to represent that the maximum over that choice istaken rather than the expected value given some policy. The backup diagram on the left
graphically represents the Bellman optimality equation (3.19) and the backup diagramon the right graphically represents (3.20).

(v⇤ )

s

(q⇤ )

max

s, a
r
s0

a
r

max

s

0

a0

Figure 3.4: Backup diagrams for v⇤ and q⇤

For finite MDPs, the Bellman optimality equation for v⇤ (3.19) has a unique solution.
The Bellman optimality equation is actually a system of equations, one for each state, so
if there are n states, then there are n equations in n unknowns. If the dynamics p of theenvironment are known, then in principle one can solve this system of equations for v⇤
using any one of a variety of methods for solving systems of nonlinear equations. One
can solve a related set of equations for q⇤ .
Once one has v⇤ , it is relatively easy to determine an optimal policy. For each states, there will be one or more actions at which the maximum is obtained in the Bellman
optimality equation. Any policy that assigns nonzero probability only to these actions is
an optimal policy. You can think of this as a one-step search. If you have the optimalvalue function, v⇤ , then the actions that appear best after a one-step search will be optimal
actions. Another way of saying this is that any policy that is greedy with respect to the
optimal evaluation function v⇤ is an optimal policy. The term greedy is used in computerscience to describe any search or decision procedure that selects alternatives based only
on local or immediate considerations, without considering the possibility that such a
selection may prevent future access to even better alternatives. Consequently, it describes
policies that select actions based only on their short-term consequences. The beauty of v⇤
is that if one uses it to evaluate the short-term consequences of actions—specifically, theone-step consequences—then a greedy policy is actually optimal in the long-term sense inwhich we are interested because v⇤ already takes into account the reward consequences ofall possible future behavior. By means of v⇤ , the optimal expected long-term return is3.6. Optimal Policies and Optimal Value Functions

65

turned into a quantity that is locally and immediately available for each state. Hence, aone-step-ahead search yields the long-term optimal actions.
Having q⇤ makes choosing optimal actions even easier. With q⇤ , the agent does noteven have to do a one-step-ahead search: for any state s, it can simply find any actionthat maximizes q⇤ (s, a). The action-value function e↵ectively caches the results of all
one-step-ahead searches. It provides the optimal expected long-term return as a valuethat is locally and immediately available for each state–action pair. Hence, at the cost ofrepresenting a function of state–action pairs, instead of just of states, the optimal actionvalue function allows optimal actions to be selected without having to know anything
about possible successor states and their values, that is, without having to know anything
about the environment’s dynamics.
Example 3.8: Solving the Gridworld Suppose we solve the Bellman equation for v⇤
for the simple grid task introduced in Example 3.5 and shown again in Figure 3.5 (left).
Recall that state A is followed by a reward of +10 and transition to state A0 , while state
B is followed by a reward of +5 and transition to state B0 . Figure 3.5 (middle) shows the
optimal value function, and Figure 3.5 (right) shows the corresponding optimal policies.
Where there are multiple arrows in a cell, all of the corresponding actions are optimal.

A

+10

B

22.0 24.4 22.0 19.4 17.5

+5

19.8 22.0 19.8 17.8 16.0

B'

17.8 19.8 17.8 16.0 14.4
16.0 17.8 16.0 14.4 13.0

A'

14.4 16.0 14.4 13.0 11.7a)Gridworld
gridworld

π⇤*
c)⇡!*

V*
b)vv
⇤*

Figure 3.5: Optimal solutions to the gridworld example.

Example 3.9: Bellman Optimality Equations for the Recycling Robot Using(3.19), we can explicitly give the Bellman optimality equation for the recycling robot
example. To make things more compact, we abbreviate the states high and low, and theactions search, wait, and recharge respectively by h, l, s, w, and re. Because there are
only two states, the Bellman optimality equation consists of two equations. The equationfor v⇤ (h) can be written as follows:v⇤ (h)

=
=
=

max
max
max

⇢

p(h|h, s)[r(h, s, h) + v⇤ (h)] + p(l|h, s)[r(h, s, l) + v⇤ (l)],p(h|h, w)[r(h, w, h) + v⇤ (h)] + p(l|h, w)[r(h, w, l) + v⇤ (l)]⇢rs + [↵v⇤ (h) + (1rw + v⇤ (h)⇢↵[rs + v⇤ (h)] + (1 ↵)[rs + v⇤ (l)],1[rw + v⇤ (h)] + 0[rw + v⇤ (l)]↵)v⇤ (l)],.

66

Chapter 3: Finite Markov Decision Processes

Following the same procedure for v⇤ (l) yields the equation8<

rs 3(1) + [(1rw + v⇤ (l),v⇤ (l) = max:v⇤ (h)

9)v⇤ (h) + v⇤ (l)], =;.

For any choice of rs , rw , ↵, , and , with 0  < 1, 0  ↵,  1, there is exactly
one pair of numbers, v⇤ (h) and v⇤ (l), that simultaneously satisfy these two nonlinear
equations.
Explicitly solving the Bellman optimality equation provides one route to finding an
optimal policy, and thus to solving the reinforcement learning problem. However, thissolution is rarely directly useful. It is akin to an exhaustive search, looking ahead at
all possibilities, computing their probabilities of occurrence and their desirabilities interms of expected rewards. This solution relies on at least three assumptions that are
rarely true in practice: (1) the dynamics of the environment are accurately known; (2)computational resources are sufficient to complete the calculation; and (3) the states
have the Markov property. For the kinds of tasks in which we are interested, one is
generally not able to implement this solution exactly because various combinations ofthese assumptions are violated. For example, although the first and third assumptions
present no problems for the game of backgammon, the second is a major impediment.
Because the game has about 1020 states, it would take thousands of years on today’s
fastest computers to solve the Bellman equation for v⇤ , and the same is true for findingq⇤ . In reinforcement learning one typically has to settle for approximate solutions.
Many di↵erent decision-making methods can be viewed as ways of approximately
solving the Bellman optimality equation. For example, heuristic search methods can beviewed as expanding the right-hand side of (3.19) several times, up to some depth, forming
a “tree” of possibilities, and then using a heuristic evaluation function to approximatev⇤ at the “leaf” nodes. (Heuristic search methods such as A⇤ are almost always basedon the episodic case.) The methods of dynamic programming can be related even moreclosely to the Bellman optimality equation. Many reinforcement learning methods can
be clearly understood as approximately solving the Bellman optimality equation, using
actual experienced transitions in place of knowledge of the expected transitions. We
consider a variety of such methods in the following chapters.
Exercise 3.20 Draw or describe the optimal state-value function for the golf example. ⇤Exercise 3.21 Draw or describe the contours of the optimal action-value function for
putting, q⇤ (s, putter), for the golf example.
⇤
Exercise 3.22 Consider the continuing MDP shown to theright. The only decision to be made is that in the top state,
where two actions are available, left and right. The numbers
show the rewards that are received deterministically after
each action. There are exactly two deterministic policies,⇡left and ⇡right . What policy is optimal if = 0? If = 0.9?
If = 0.5?
⇤

0

left

right

+1

0

+2

3.7. Optimality and Approximation

67

Exercise 3.23 Give the Bellman equation for q⇤ for the recycling robot.

⇤

Exercise 3.24 Figure 3.5 gives the optimal value of the best state of the gridworld as
24.4, to one decimal place. Use your knowledge of the optimal policy and (3.8) to express
this value symbolically, and then to compute it to three decimal places.
⇤
Exercise 3.25 Give an equation for v⇤ in terms of q⇤ .
Exercise 3.26 Give an equation for q⇤ in terms of v⇤ and the four-argument p.
Exercise 3.27 Give an equation for ⇡⇤ in terms of q⇤ .
Exercise 3.28 Give an equation for ⇡⇤ in terms of v⇤ and the four-argument p.

⇤
⇤
⇤
⇤

Exercise 3.29 Rewrite the four Bellman equations for the four value functions (v⇡ , v⇤ , q⇡ ,and q⇤ ) in terms of the three argument function p (3.4) and the two-argument function r(3.5).
⇤

3.7

Optimality and Approximation

We have defined optimal value functions and optimal policies. Clearly, an agent thatlearns an optimal policy has done very well, but in practice this rarely happens. For
the kinds of tasks in which we are interested, optimal policies can be generated onlywith extreme computational cost. A well-defined notion of optimality organizes theapproach to learning we describe in this book and provides a way to understand thetheoretical properties of various learning algorithms, but it is an ideal that agents canonly approximate. As we discussed above, even if we have a complete and accurate modelof the environment’s dynamics, it is usually not possible to simply compute an optimal
policy by solving the Bellman optimality equation. For example, board games such as
chess are a tiny fraction of human experience, yet large, custom-designed computers stillcannot compute the optimal moves. A critical aspect of the problem facing the agent isalways the computational power available to it, in particular, the amount of computation
it can perform in a single time step.
The memory available is also an important constraint. A large amount of memory
is often required to build up approximations of value functions, policies, and models.
In tasks with small, finite state sets, it is possible to form these approximations usingarrays or tables with one entry for each state (or state–action pair). This we call thetabular case, and the corresponding methods we call tabular methods. In many casesof practical interest, however, there are far more states than could possibly be entries
in a table. In these cases the functions must be approximated, using some sort of morecompact parameterized function representation.
Our framing of the reinforcement learning problem forces us to settle for approximations. However, it also presents us with some unique opportunities for achieving
useful approximations. For example, in approximating optimal behavior, there may be
many states that the agent faces with such a low probability that selecting suboptimal
actions for them has little impact on the amount of reward the agent receives. Tesauro’s
backgammon player, for example, plays with exceptional skill even though it might make

68

Chapter 3: Finite Markov Decision Processes

very bad decisions on board configurations that never occur in games against experts. In
fact, it is possible that TD-Gammon makes bad decisions for a large fraction of the game’s
state set. The online nature of reinforcement learning makes it possible to approximate
optimal policies in ways that put more e↵ort into learning to make good decisions forfrequently encountered states, at the expense of less e↵ort for infrequently encounteredstates. This is one key property that distinguishes reinforcement learning from otherapproaches to approximately solving MDPs.

3.8

Summary

Let us summarize the elements of the reinforcement learning problem that we havepresented in this chapter. Reinforcement learning is about learning from interaction
how to behave in order to achieve a goal. The reinforcement learning agent and its
environment interact over a sequence of discrete time steps. The specification of theirinterface defines a particular task: the actions are the choices made by the agent; thestates are the basis for making the choices; and the rewards are the basis for evaluating
the choices. Everything inside the agent is known and controllable. Its environment, on
the other hand, is incompletely controllable and may or may not be completely known.
A policy is a stochastic rule by which the agent selects actions as a function of states.
The agent’s objective is to maximize the amount of reward it receives over time.
When the reinforcement learning setup described above is formulated with well definedtransition probabilities it constitutes a Markov decision process (MDP). A finite MDP isan MDP with finite state, action, and (as we formulate it here) reward sets. Much of thecurrent theory of reinforcement learning is restricted to finite MDPs, but the methods
and ideas apply more generally.
The return is the function of future rewards that the agent seeks to maximize (inexpected value). It has several di↵erent definitions depending upon the nature of thetask and whether one wishes to discount delayed reward. The undiscounted formulation
is appropriate for episodic tasks, in which the agent–environment interaction breaks
naturally into episodes; the discounted formulation is appropriate for tabular continuingtasks, in which the interaction does not naturally break into episodes but continues
without limit (but see Sections 10.3–4). We try to define the returns for the two kinds oftasks such that one set of equations can apply to both the episodic and continuing cases.
A policy’s value functions (v⇡ and q⇡ ) assign to each state, or state–action pair, theexpected return from that state, or state–action pair, given that the agent uses thepolicy. The optimal value functions (v⇤ and q⇤ ) assign to each state, or state–action pair,the largest expected return achievable by any policy. A policy whose value functions
are optimal is an optimal policy. Whereas the optimal value functions for states andstate–action pairs are unique for a given MDP, there can be many optimal policies. Any
policy that is greedy with respect to the optimal value functions must be an optimal
policy. The Bellman optimality equations are special consistency conditions that the
optimal value functions must satisfy and that can, in principle, be solved for the optimalvalue functions, from which an optimal policy can be determined with relative ease.

3.8. Summary

69

A reinforcement learning problem can be posed in a variety of di↵erent ways depending
on assumptions about the level of knowledge initially available to the agent. In problemsof complete knowledge, the agent has a complete and accurate model of the environment’s
dynamics. If the environment is an MDP, then such a model consists of the complete fourargument dynamics function p (3.2). In problems of incomplete knowledge, a complete
and perfect model of the environment is not available.
Even if the agent had a complete and accurate environment model, the agent would
typically be unable to fully use it because of limitations on its memory and computation
per time step. In particular, extensive memory may be required to build up accurate
approximations of value functions, policies, and models. In most cases of practical interest
there are far more states than could possibly be entries in a table, and approximations
must be made.
A well-defined notion of optimality organizes the approach to learning we describe inthis book and provides a way to understand the theoretical properties of various learningalgorithms, but it is an ideal that reinforcement learning agents can only approximate
to varying degrees. In reinforcement learning we are very much concerned with cases inwhich optimal solutions cannot be found but must be approximated in some way.

Bibliographical and Historical Remarks
The reinforcement learning problem is deeply indebted to the idea of Markov decision
processes (MDPs) from the field of optimal control. These historical influences and othermajor influences from psychology are described in the brief history given in Chapter 1.
Reinforcement learning adds to MDPs a focus on approximation and incomplete information for realistically large problems. MDPs and the reinforcement learning problem
are only weakly linked to traditional learning and decision-making problems in artificial
intelligence. However, artificial intelligence is now vigorously exploring MDP formulations
for planning and decision making from a variety of perspectives. MDPs are more generalthan previous formulations used in artificial intelligence in that they permit more generalkinds of goals and uncertainty.
The theory of MDPs is treated by, for example, Bertsekas (2005), White (1969), Whittle(1982, 1983), and Puterman (1994). A particularly compact treatment of the finite caseis given by Ross (1983). MDPs are also studied under the heading of stochastic optimal
control, where adaptive optimal control methods are most closely related to reinforcement
learning (e.g., Kumar, 1985; Kumar and Varaiya, 1986).
The theory of MDPs evolved from e↵orts to understand the problem of making sequencesof decisions under uncertainty, where each decision can depend on the previous decisions
and their outcomes. It is sometimes called the theory of multistage decision processes,
or sequential decision processes, and has roots in the statistical literature on sequential
sampling beginning with the papers by Thompson (1933, 1934) and Robbins (1952) that
we cited in Chapter 2 in connection with bandit problems (which are prototypical MDPs
if formulated as multiple-situation problems).
The earliest instance (that we are aware of) in which reinforcement learning was
discussed using the MDP formalism is Andreae’s (1969) description of a unified view of70

Chapter 3: Finite Markov Decision Processes

learning machines. Witten and Corbin (1973) experimented with a reinforcement learning
system later analyzed by Witten (1977, 1976a) using the MDP formalism. Although
he did not explicitly mention MDPs, Werbos (1977) suggested approximate solution
methods for stochastic optimal control problems that are related to modern reinforcement
learning methods (see also Werbos, 1982, 1987, 1988, 1989, 1992). Although Werbos’s
ideas were not widely recognized at the time, they were prescient in emphasizing theimportance of approximately solving optimal control problems in a variety of domains,
including artificial intelligence. The most influential integration of reinforcement learning
and MDPs is due to Watkins (1989).
3.1

Our characterization of the dynamics of an MDP in terms of p(s0 , r |s, a) isslightly unusual. It is more common in the MDP literature to describe thedynamics in terms of the state transition probabilities p(s0 |s, a) and expected
next rewards r(s, a). In reinforcement learning, however, we more often haveto refer to individual actual or sample rewards (rather than just their expectedvalues). Our notation also makes it plainer that St and Rt are in general jointly
determined, and thus must have the same time index. In teaching reinforcement
learning, we have found our notation to be more straightforward conceptuallyand easier to understand.
For a good intuitive discussion of the system-theoretic concept of state, see
Minsky (1967).
The bioreactor example is based on the work of Ungar (1990) and Miller andWilliams (1992). The recycling robot example was inspired by the can-collecting
robot built by Jonathan Connell (1989). Kober and Peters (2012) present acollection of robotics applications of reinforcement learning.

3.2

An explicit statement of the reward hypothesis was suggested by Michael Littman(personal communication).

3.3–4 The terminology of episodic and continuing tasks is di↵erent from that usuallyused in the MDP literature. In that literature it is common to distinguishthree types of tasks: (1) finite-horizon tasks, in which interaction terminates
after a particular fixed number of time steps; (2) indefinite-horizon tasks, inwhich interaction can last arbitrarily long but must eventually terminate; and(3) infinite-horizon tasks, in which interaction does not terminate. Our episodic
and continuing tasks are similar to indefinite-horizon and infinite-horizon tasks,
respectively, but we prefer to emphasize the di↵erence in the nature of theinteraction. This di↵erence seems more fundamental than the di↵erence in theobjective functions emphasized by the usual terms. Often episodic tasks use
an indefinite-horizon objective function and continuing tasks an infinite-horizon
objective function, but we see this as a common coincidence rather than afundamental di↵erence.
The pole-balancing example is from Michie and Chambers (1968) and Barto,
Sutton, and Anderson (1983).

3.8. Summary71

3.5–6 Assigning value on the basis of what is good or bad in the long run has ancientroots. In control theory, mapping states to numerical values representing thelong-term consequences of control decisions is a key part of optimal control theory,
which was developed in the 1950s by extending nineteenth century state-function
theories of classical mechanics (see, for example, Schultz and Melsa, 1967). In
describing how a computer could be programmed to play chess, Shannon (1950)
suggested using an evaluation function that took into account the long-term
advantages and disadvantages of chess positions.
Watkins’s (1989) Q-learning algorithm for estimating q⇤ (Chapter 6) made actionvalue functions an important part of reinforcement learning, and consequently
these functions are often called “Q-functions.” But the idea of an action-valuefunction is much older than this. Shannon (1950) suggested that a function
h(P, M ) could be used by a chess-playing program to decide whether a move M
in position P is worth exploring. Michie’s (1961, 1963) MENACE system andMichie and Chambers’s (1968) BOXES system can be understood as estimating
action-value functions. In classical physics, Hamilton’s principal function isan action-value function; Newtonian dynamics are greedy with respect to thisfunction (e.g., Goldstein, 1957). Action-value functions also played a central rolein Denardo’s (1967) theoretical treatment of dynamic programming in terms ofcontraction mappings.
The Bellman optimality equation (for v⇤ ) was popularized by Richard Bellman(1957a), who called it the “basic functional equation.” The counterpart of the
Bellman optimality equation for continuous time and state problems is known
as the Hamilton–Jacobi–Bellman equation (or often just the Hamilton–Jacobi
equation), indicating its roots in classical physics (e.g., Schultz and Melsa, 1967).
The golf example was suggested by Chris Watkins.

Chapter 4

Dynamic Programming
The term dynamic programming (DP) refers to a collection of algorithms that can beused to compute optimal policies given a perfect model of the environment as a Markov
decision process (MDP). Classical DP algorithms are of limited utility in reinforcement
learning both because of their assumption of a perfect model and because of their greatcomputational expense, but they are still important theoretically. DP provides an essential
foundation for the understanding of the methods presented in the rest of this book. In
fact, all of these methods can be viewed as attempts to achieve much the same e↵ect as
DP, only with less computation and without assuming a perfect model of the environment.
Starting with this chapter, we usually assume that the environment is a finite MDP.
That is, we assume that its state, action, and reward sets, S, A, and R, are finite, andthat its dynamics are given by a set of probabilities p(s0 , r |s, a), for all s 2 S, a 2 A(s),r 2 R, and s0 2 S+ (S+ is S plus a terminal state if the problem is episodic). Although
DP ideas can be applied to problems with continuous state and action spaces, exact
solutions are possible only in special cases. A common way of obtaining approximate
solutions for tasks with continuous states and actions is to quantize the state and action
spaces and then apply finite-state DP methods. The methods we explore in Part II are
applicable to continuous problems and are a significant extension of that approach.
The key idea of DP, and of reinforcement learning generally, is the use of value functions
to organize and structure the search for good policies. In this chapter we show how DP
can be used to compute the value functions defined in Chapter 3. As discussed there, we
can easily obtain optimal policies once we have found the optimal value functions, v⇤ orq⇤ , which satisfy the Bellman optimality equations:v⇤ (s) = max E[Rt+1 + v⇤ (St+1 ) | St = s, At = a]
a
h
i
X
= max
p(s0 , r |s, a) r + v⇤ (s0 ) ,
a

or

(4.1)

s0 ,r

h
i
0
q⇤ (s, a) = E Rt+1 + max
q
(S
,
a
)
S
=
s,
A
=
a
⇤
t+1
t
t
a0
h
i
X
0
=
p(s , r |s, a) r + maxq⇤ (s0 , a0 ) ,
0
s0 ,r

a

(4.2)

74

Chapter 4: Dynamic Programming

for all s 2 S, a 2 A(s), and s0 2 S+ . As we shall see, DP algorithms are obtained byturning Bellman equations such as these into assignments, that is, into update rules for
improving approximations of the desired value functions.

4.1

Policy Evaluation (Prediction)

First we consider how to compute the state-value function v⇡ for an arbitrary policy ⇡.
This is called policy evaluation in the DP literature. We also refer to it as the prediction
problem. Recall from Chapter 3 that, for all s 2 S,.
v⇡ (s) = E⇡[Gt | St = s]

= E⇡[Rt+1 + Gt+1 | St = s]

= E⇡[Rt+1 + v⇡ (St+1 ) | St = s]
h
i
X
X
=
⇡(a|s)p(s0 , r |s, a) r + v⇡ (s0 ) ,(from (3.9))
(4.3)
(4.4)

s0 ,r

a

where ⇡(a|s) is the probability of taking action a in state s under policy ⇡, and the
expectations are subscripted by ⇡ to indicate that they are conditional on ⇡ being followed.
The existence and uniqueness of v⇡ are guaranteed as long as either < 1 or eventual
termination is guaranteed from all states under the policy ⇡.
If the environment’s dynamics are completely known, then (4.4) is a system of |S|simultaneous linear equations in |S| unknowns (the v⇡ (s), s 2 S). In principle, its solution
is a straightforward, if tedious, computation. For our purposes, iterative solution methods
are most suitable. Consider a sequence of approximate value functions v0 , v1 , v2 , . . .,each mapping S+ to R (the real numbers). The initial approximation, v0 , is chosen
arbitrarily (except that the terminal state, if any, must be given value 0), and eachsuccessive approximation is obtained by using the Bellman equation for v⇡ (4.4) as an
update rule:.
vk+1 (s) = E⇡[Rt+1 + vk (St+1 ) | St = s]
h
i
X
X
=
⇡(a|s)p(s0 , r |s, a) r + vk (s0 ) ,
(4.5)
a

s0 ,r

for all s 2 S. Clearly, vk = v⇡ is a fixed point for this update rule because the Bellman
equation for v⇡ assures us of equality in this case. Indeed, the sequence {vk } can beshown in general to converge to v⇡ as k ! 1 under the same conditions that guarantee
the existence of v⇡ . This algorithm is called iterative policy evaluation.
To produce each successive approximation, vk+1 from vk , iterative policy evaluation
applies the same operation to each state s: it replaces the old value of s with a new value
obtained from the old values of the successor states of s, and the expected immediate
rewards, along all the one-step transitions possible under the policy being evaluated. We
call this kind of operation an expected update. Each iteration of iterative policy evaluation
updates the value of every state once to produce the new approximate value function

4.1. Policy Evaluation (Prediction)75

vk+1 . There are several di↵erent kinds of expected updates, depending on whether astate (as here) or a state–action pair is being updated, and depending on the precise way
the estimated values of the successor states are combined. All the updates done in DP
algorithms are called expected updates because they are based on an expectation over all
possible next states rather than on a sample next state. The nature of an update canbe expressed in an equation, as above, or in a backup diagram like those introduced in
Chapter 3. For example, the backup diagram corresponding to the expected update usedin iterative policy evaluation is shown on page 59.
To write a sequential computer program to implement iterative policy evaluation as
given by (4.5) you would have to use two arrays, one for the old values, vk (s), and one
for the new values, vk+1 (s). With two arrays, the new values can be computed one by
one from the old values without the old values being changed. Alternatively, you could
use one array and update the values “in place,” that is, with each new value immediately
overwriting the old one. Then, depending on the order in which the states are updated,sometimes new values are used instead of old ones on the right-hand side of (4.5). This
in-place algorithm also converges to v⇡ ; in fact, it usually converges faster than the
two-array version, as you might expect, because it uses new data as soon as they areavailable. We think of the updates as being done in a sweep through the state space. For
the in-place algorithm, the order in which states have their values updated during thesweep has a significant influence on the rate of convergence. We usually have the in-placeversion in mind when we think of DP algorithms.
A complete in-place version of iterative policy evaluation is shown in pseudocode in
the box below. Note how it handles termination. Formally, iterative policy evaluation
converges only in the limit, but in practice it must be halted short of this. The pseudocode
tests the quantity maxs2S |vk+1 (s) vk (s)| after each sweep and stops when it is sufficiently
small.
Iterative Policy Evaluation, for estimating V ⇡ v⇡
Input ⇡, the policy to be evaluatedAlgorithm parameter: a small threshold ✓ > 0 determining accuracy of estimation
Initialize V (s) arbitrarily, for s 2 S, and V (terminal) to 0
Loop:
0
Loop for each s 2 S:
v
V (s)
⇥
⇤
P
P
0
0
V (s)
a ⇡(a|s)s0 ,r p(s , r |s, a) r + V (s )max( , |v V (s)|)
until < ✓76

Chapter 4: Dynamic Programming

Example 4.1 Consider the 4⇥4 gridworld shown below.

actions

1

2

3

4

5

6

7

8

9

10

11

12

13

14

Rtr =
= !1 1on all transitions

The nonterminal states are S = {1, 2, . . . , 14}. There are four actions possible in eachstate, A = {up, down, right, left}, which deterministically cause the corresponding
state transitions, except that actions that would take the agent o↵ the grid in fact leave
the state unchanged. Thus, for instance, p(6, 1|5, right) = 1, p(7, 1|7, right) = 1,
and p(10, r |5, right) = 0 for all r 2 R. This is an undiscounted, episodic task. The
reward is 1 on all transitions until the terminal state is reached. The terminal state isshaded in the figure (although it is shown in two places, it is formally one state). The
expected reward function is thus r(s, a, s0 ) = 1 for all states s, s0 and actions a. Suppose
the agent follows the equiprobable random policy (all actions equally likely). The left sideof Figure 4.1 shows the sequence of value functions {vk } computed by iterative policy
evaluation. The final estimate is in fact v⇡ , which in this case gives for each state thenegation of the expected number of steps from that state until termination.
Exercise 4.1 In Example 4.1, if ⇡ is the equiprobable random policy, what is q⇡ (11, down)?
What is q⇡ (7, down)?
⇤
Exercise 4.2 In Example 4.1, suppose a new state 15 is added to the gridworld just below
state 13, and its actions, left, up, right, and down, take the agent to states 12, 13, 14,
and 15, respectively. Assume that the transitions from the original states are unchanged.
What, then, is v⇡ (15) for the equiprobable random policy? Now suppose the dynamics ofstate 13 are also changed, such that action down from state 13 takes the agent to the newstate 15. What is v⇡ (15) for the equiprobable random policy in this case?
⇤
Exercise 4.3 What are the equations analogous to (4.3), (4.4), and (4.5), but for actionvalue functions instead of state-value functions?
⇤

4.2

Policy Improvement

Our reason for computing the value function for a policy is to help find better policies.
Suppose we have determined the value function v⇡ for an arbitrary deterministic policy⇡. For some state s we would like to know whether or not we should change the policy
to deterministically choose an action a 6= ⇡(s). We know how good it is to follow thecurrent policy from s—that is v⇡ (s)—but would it be better or worse to change to thenew policy? One way to answer this question is to consider selecting a in s and thereafter4.2. Policy Improvement

77

Vvkk for the
random policy
RandomPolicyGreedyPolicygreedy policy
vvkk
w.r.t. V
w.r.t.

0.0 0.0 0.0 0.0

k=0

0.0 0.0 0.0 0.0
0.0 0.0 0.0 0.0

randompolicy0.0 0.0 0.0 0.0

0.0 -1.0 -1.0 -1.0

k=1-1.0 -1.0 -1.0 -1.0
-1.0 -1.0 -1.0 -1.0
-1.0 -1.0 -1.0 0.0

0.0 -1.7 -2.0 -2.0

k=2-1.7 -2.0 -2.0 -2.0
-2.0 -2.0 -2.0 -1.7
-2.0 -2.0 -1.7 0.0

0.0 -2.4 -2.9 -3.0

k=3-2.4 -2.9 -3.0 -2.9
-2.9 -3.0 -2.9 -2.4
-3.0 -2.9 -2.4 0.0

0.0 -6.1 -8.4 -9.0

k = 10-6.1 -7.7 -8.4 -8.4
-8.4 -8.4 -7.7 -6.1

optimalpolicy-9.0 -8.4 -6.1 0.0

0.0 -14. -20. -22.

k=!

-14. -18. -20. -20.
-20. -20. -18. -14.
-22. -20. -14. 0.0

Figure 4.1: Convergence of iterative policy evaluation on a small gridworld. The left column isthe sequence of approximations of the state-value function for the random policy (all actions
equally likely). The right column is the sequence of greedy policies corresponding to the valuefunction estimates (arrows are shown for all actions achieving the maximum, and the numbers
shown are rounded to two significant digits). The last policy is guaranteed only to be an
improvement over the random policy, but in this case it, and all policies after the third iteration,
are optimal.

78

Chapter 4: Dynamic Programming

following the existing policy, ⇡. The value of this way of behaving is.
q⇡ (s, a) = E[Rt+1 + v⇡ (St+1 ) | St = s, At = a]hiX=p(s0 , r |s, a) r + v⇡ (s0 ) .

(4.6)

s0 ,r

The key criterion is whether this is greater than or less than v⇡ (s). If it is greater—that
is, if it is better to select a once in s and thereafter follow ⇡ than it would be to follow⇡ all the time—then one would expect it to be better still to select a every time s is
encountered, and that the new policy would in fact be a better one overall.
That this is true is a special case of a general result called the policy improvement
theorem. Let ⇡ and ⇡ 0 be any pair of deterministic policies such that, for all s 2 S,q⇡ (s, ⇡ 0 (s))

v⇡ (s).

(4.7)

Then the policy ⇡ 0 must be as good as, or better than, ⇡. That is, it must obtain greateror equal expected return from all states s 2 S:v⇡0 (s)v⇡ (s).

(4.8)

Moreover, if there is strict inequality of (4.7) at any state, then there must be strict
inequality of (4.8) at that state.
The policy improvement theorem applies to the two policies that we considered at the
beginning of this section: an original deterministic policy, ⇡, and a changed policy, ⇡ 0 ,that is identical to ⇡ except that ⇡ 0 (s) = a 6= ⇡(s). For states other than s, (4.7) holds
because the two sides are equal. Thus, if q⇡ (s, a) > v⇡ (s), then the changed policy is
indeed better than ⇡.
The idea behind the proof of the policy improvement theorem is easy to understand.
Starting from (4.7), we keep expanding the q⇡ side with (4.6) and reapplying (4.7) until
we get v⇡0 (s):v⇡ (s)  q⇡ (s, ⇡ 0 (s))= E[Rt+1 + v⇡ (St+1 ) | St = s, At = ⇡ 0 (s)](by (4.6))= E⇡0[Rt+1 + v⇡ (St+1 ) | St = s]

 E⇡0[Rt+1 + q⇡ (St+1 , ⇡ 0 (St+1 )) | St = s]..
.

(by (4.7))0= E⇡0[Rt+1 + E[Rt+2 + v⇡ (St+2 )|St+1 , At+1 = ⇡ (St+1 )] | St = s]⇥⇤
= E⇡0 Rt+1 + Rt+2 + 2 v⇡ (St+2 ) St = s⇥⇤ E⇡0 Rt+1 + Rt+2 + 2 Rt+3 + 3 v⇡ (St+3 ) St = s⇥ E⇡0 Rt+1 + Rt+2 += v⇡0 (s).

2

Rt+3 +

3

Rt+4 + · · ·St = s

⇤So far we have seen how, given a policy and its value function, we can easily evaluatea change in the policy at a single state. It is a natural extension to consider changes at

4.2. Policy Improvement79

all states, selecting at each state the action that appears best according to q⇡ (s, a). In
other words, to consider the new greedy policy, ⇡ 0 , given by⇡ 0 (s).
=argmax q⇡ (s, a)a==argmax E[Rt+1 + v⇡ (St+1 ) | St = s, At = a]ahiXargmaxp(s0 , r |s, a) r + v⇡ (s0 ) ,
a

(4.9)

s0 ,r

where argmaxa denotes the value of a at which the expression that follows is maximized(with ties broken arbitrarily). The greedy policy takes the action that looks best in the
short term—after one step of lookahead—according to v⇡ . By construction, the greedy
policy meets the conditions of the policy improvement theorem (4.7), so we know that it
is as good as, or better than, the original policy. The process of making a new policy that
improves on an original policy, by making it greedy with respect to the value function of
the original policy, is called policy improvement.
Suppose the new greedy policy, ⇡ 0 , is as good as, but not better than, the old policy ⇡.
Then v⇡ = v⇡0 , and from (4.9) it follows that for all s 2 S:v⇡0 (s)==max E[Rt+1 + v⇡0 (St+1 ) | St = s, At = a]
a
h
i
X
max
p(s0 , r |s, a) r + v⇡0 (s0 ) .
a

s0 ,r

But this is the same as the Bellman optimality equation (4.1), and therefore, v⇡0 must bev⇤ , and both ⇡ and ⇡ 0 must be optimal policies. Policy improvement thus must give us astrictly better policy except when the original policy is already optimal.
So far in this section we have considered the special case of deterministic policies.
In the general case, a stochastic policy ⇡ specifies probabilities, ⇡(a|s), for taking eachaction, a, in each state, s. We will not go through the details, but in fact all the ideas ofthis section extend easily to stochastic policies. In particular, the policy improvement
theorem carries through as stated for the stochastic case. In addition, if there are ties inpolicy improvement steps such as (4.9)—that is, if there are several actions at which the
maximum is achieved—then in the stochastic case we need not select a single action fromamong them. Instead, each maximizing action can be given a portion of the probabilityof being selected in the new greedy policy. Any apportioning scheme is allowed as longas all submaximal actions are given zero probability.
The last row of Figure 4.1 shows an example of policy improvement for stochastic
policies. Here the original policy, ⇡, is the equiprobable random policy, and the newpolicy, ⇡ 0 , is greedy with respect to v⇡ . The value function v⇡ is shown in the bottom-leftdiagram and the set of possible ⇡ 0 is shown in the bottom-right diagram. The stateswith multiple arrows in the ⇡ 0 diagram are those in which several actions achieve themaximum in (4.9); any apportionment of probability among these actions is permitted.
For any such policy, its state values v⇡0 (s) can be seen by inspection to be either 1, 2,
or 3, for all states s 2 S, whereas v⇡ (s) is at most 14. Thus, v⇡0 (s) v⇡ (s), for all

80

Chapter 4: Dynamic Programmings 2 S, illustrating policy improvement. Although in this case the new policy ⇡ 0 happens
to be optimal, in general only an improvement is guaranteed.

4.3

Policy Iteration

Once a policy, ⇡, has been improved using v⇡ to yield a better policy, ⇡ 0 , we can thencompute v⇡0 and improve it again to yield an even better ⇡ 00 . We can thus obtain asequence of monotonically improving policies and value functions:
E

I

E

I

E

I

E

⇡0 ! v ⇡ 0 ! ⇡1 ! v ⇡ 1 ! ⇡2 ! · · · ! ⇡⇤ ! v ⇤ ,EIwhere ! denotes a policy evaluation and ! denotes a policy improvement. Eachpolicy is guaranteed to be a strict improvement over the previous one (unless it is already
optimal). Because a finite MDP has only a finite number of deterministic policies, thisprocess must converge to an optimal policy and the optimal value function in a finite
number of iterations.
This way of finding an optimal policy is called policy iteration. A complete algorithm isgiven in the box below. Note that each policy evaluation, itself an iterative computation,is started with the value function for the previous policy. This typically results in a great
increase in the speed of convergence of policy evaluation (presumably because the valuefunction changes little from one policy to the next).
Policy Iteration (using iterative policy evaluation) for estimating ⇡ ⇡ ⇡⇤1. Initialization.
V (s) 2 R and ⇡(s) 2 A(s) arbitrarily for all s 2 S; V (terminal) = 0
2. Policy Evaluation
Loop:
0
Loop for each s 2 S:
v
V (s)
⇥
⇤
P
0
0
V (s)
s0 ,r p(s , r |s, ⇡(s)) r + V (s )max( , |v V (s)|)
until < ✓ (a small positive number determining the accuracy of estimation)
3. Policy Improvement
policy-stabletrue
For each s 2 S:
old-action
⇡(s)P
⇥
⇤
⇡(s)
argmaxa s0 ,r p(s0 , r |s, a) r + V (s0 )If old-action 6= ⇡(s), then policy-stablef alseIf policy-stable, then stop and return V ⇡ v⇤ and ⇡ ⇡ ⇡⇤ ; else go to 2

4.3. Policy Iteration

81

Example 4.2: Jack’s Car Rental Jack manages two locations for a nationwide car
rental company. Each day, some number of customers arrive at each location to rent cars.
If Jack has a car available, he rents it out and is credited $10 by the national company.
If he is out of cars at that location, then the business is lost. Cars become available forrenting the day after they are returned. To help ensure that cars are available where
they are needed, Jack can move them between the two locations overnight, at a cost of$2 per car moved. We assume that the number of cars requested and returned at eachlocation are Poisson random variables, meaning that the probability that the number isnn is n! e , where is the expected number. Suppose is 3 and 4 for rental requests at
the first and second locations and 3 and 2 for returns. To simplify the problem slightly,
we assume that there can be no more than 20 cars at each location (any additional cars
are returned to the nationwide company, and thus disappear from the problem) and amaximum of five cars can be moved from one location to the other in one night. We takethe discount rate to be = 0.9 and formulate this as a continuing finite MDP, where
the time steps are days, the state is the number of cars at each location at the end of
the day, and the actions are the net numbers of cars moved between the two locations
overnight. Figure 4.2 shows the sequence of policies found by policy iteration startingfrom the policy that never moves any cars.


⇡00

1
⇡

2
⇡
5

5
4

0

4

3

2

3

2

43

21

0

1

1

0
1
2
3 -4

5
3

2

1

4

3

2

2

1

612

0

0

⇡4

2
tio 0
n

4

V4 v

ca

5

⇡44

3 4

0

#Cars at first location

20

⇡33

1
2

0

#Cars at second location

20

lo
st
fir

seco

at

rs at

nd lo

catio 0
n

s

#Ca

ar

1
2
3 4

#C

0

420
0
1
2
3 4

Figure 4.2: The sequence of policies found by policy iteration on Jack’s car rental problem,
and the final state-value function. The first five diagrams show, for each number of cars at
each location at the end of the day, the number of cars to be moved from the first location to
the second (negative numbers indicate transfers from the second location to the first). Eachsuccessive policy is a strict improvement over the previous policy, and the last policy is optimal.

82

Chapter 4: Dynamic Programming

Policy iteration often converges in surprisingly few iterations, as illustrated in theexample of Jack’s car rental and in the example in Figure 4.1. The bottom-left diagram of
Figure 4.1 shows the value function for the equiprobable random policy, and the bottomright diagram shows a greedy policy for this value function. The policy improvement
theorem assures us that these policies are better than the original random policy. In thiscase, however, these policies are not just better, but optimal, proceeding to the terminal
states in the minimum number of steps. In this example, policy iteration would find the
optimal policy after just one iteration.
Exercise 4.4 The policy iteration algorithm on page 80 has a subtle bug in that it may
never terminate if the policy continually switches between two or more policies that are
equally good. This is okay for pedagogy, but not for actual use. Modify the pseudocode
so that convergence is guaranteed.
⇤
Exercise 4.5 How would policy iteration be defined for action values? Give a complete
algorithm for computing q⇤ , analogous to that on page 80 for computing v⇤ . Please pay
special attention to this exercise, because the ideas involved will be used throughout therest of the book.
⇤
Exercise 4.6 Suppose you are restricted to considering only policies that are "-soft,
meaning that the probability of selecting each action in each state, s, is at least "/|A(s)|.
Describe qualitatively the changes that would be required in each of the steps 3, 2, and 1,
in that order, of the policy iteration algorithm for v⇤ on page 80.
⇤
Exercise 4.7 (programming) Write a program for policy iteration and re-solve Jack’s car
rental problem with the following changes. One of Jack’s employees at the first location
rides a bus home each night and lives near the second location. She is happy to shuttle
one car to the second location for free. Each additional car still costs $2, as do all cars
moved in the other direction. In addition, Jack has limited parking space at each location.
If more than 10 cars are kept overnight at a location (after any moving of cars), then an
additional cost of $4 must be incurred to use a second parking lot (independent of how
many cars are kept there). These sorts of nonlinearities and arbitrary dynamics often
occur in real problems and cannot easily be handled by optimization methods other than
dynamic programming. To check your program, first replicate the results given for theoriginal problem.
⇤

4.4

Value Iteration

One drawback to policy iteration is that each of its iterations involves policy evaluation,
which may itself be a protracted iterative computation requiring multiple sweeps throughthe state set. If policy evaluation is done iteratively, then convergence exactly to v⇡
occurs only in the limit. Must we wait for exact convergence, or can we stop short ofthat? The example in Figure 4.1 certainly suggests that it may be possible to truncate
policy evaluation. In that example, policy evaluation iterations beyond the first three
have no e↵ect on the corresponding greedy policy.
In fact, the policy evaluation step of policy iteration can be truncated in several ways
without losing the convergence guarantees of policy iteration. One important special

4.4. Value Iteration

83

case is when policy evaluation is stopped after just one sweep (one update of each state).
This algorithm is called value iteration. It can be written as a particularly simple update
operation that combines the policy improvement and truncated policy evaluation steps:
vk+1 (s).
==max E[Rt+1 + vk (St+1 ) | St = s, At = a]
a
h
i
X
max
p(s0 , r |s, a) r + vk (s0 ) ,a(4.10)

s0 ,rfor all s 2 S. For arbitrary v0 , the sequence {vk } can be shown to converge to v⇤ under
the same conditions that guarantee the existence of v⇤ .
Another way of understanding value iteration is by reference to the Bellman optimality
equation (4.1). Note that value iteration is obtained simply by turning the Bellman
optimality equation into an update rule. Also note how the value iteration update is
identical to the policy evaluation update (4.5) except that it requires the maximum to betaken over all actions. Another way of seeing this close relationship is to compare thebackup diagrams for these algorithms on page 59 (policy evaluation) and on the left of
Figure 3.4 (value iteration). These two are the natural backup operations for computingv⇡ and v⇤ .
Finally, let us consider how value iteration terminates. Like policy evaluation, value
iteration formally requires an infinite number of iterations to converge exactly to v⇤ . In
practice, we stop once the value function changes by only a small amount in a sweep.
The box below shows a complete algorithm with this kind of termination condition.
Value Iteration, for estimating ⇡ ⇡ ⇡⇤Algorithm parameter: a small threshold ✓ > 0 determining accuracy of estimation
Initialize V (s), for all s 2 S+ , arbitrarily except that V (terminal) = 0
Loop:
|
0
| Loop for each s 2 S:
|
v
V (s)
⇥
⇤
P
|
V (s)
maxa s0 ,r p(s0 , r |s, a) r + V (s0 )|max( , |v V (s)|)
until < ✓Output a deterministic
P policy, ⇡ ⇡ ⇡⇥⇤ , such that⇤⇡(s) = argmaxa s0 ,r p(s0 , r |s, a) r + V (s0 )Value iteration e↵ectively combines, in each of its sweeps, one sweep of policy evaluationand one sweep of policy improvement. Faster convergence is often achieved by interposing
multiple policy evaluation sweeps between each policy improvement sweep. In general,
the entire class of truncated policy iteration algorithms can be thought of as sequencesof sweeps, some of which use policy evaluation updates and some of which use value
iteration updates. Because the max operation in (4.10) is the only di↵erence between

84

Chapter 4: Dynamic Programmingthese updates, this just means that the max operation is added to some sweeps of policy
evaluation. All of these algorithms converge to an optimal policy for discounted finite
MDPs.
Example 4.3: Gambler’s Problem A gambler has the opportunity to make bets on theoutcomes of a sequence of coin flips. If the coin comes up heads, he wins as many dollars as
he has staked on that flip; if it is tails, he loses his stake. The game ends when the gambler
wins by reaching his goal of $100, or loses by running out of money. On each flip, the gambler must decide what portion of his capital to stake, in integer numbers of dollars. This
problem can be formulated as an undiscounted, episodic, finite MDP. The state is the gambler’s capital, s 2 {1, 2, . . . , 99} and the actions are stakes, a 2 {0, 1, . . . , min(s, 100 s)}.
The reward is zero on all transitions except those on which the gam1
bler reaches his goal, when it is +1.
Final value
0.8function
The state-value function then gives
the probability of winning from each
0.6
sweep 32
Value
state. A policy is a mapping from estimates
0.4
levels of capital to stakes. The optimal policy maximizes the probability
sweep 1
0.2
sweep 2of reaching the goal. Let ph denote
sweep 30
the probability of the coin coming
1
99
25
50
75
up heads. If ph is known, then the
Capitalentire problem is known and it canbe solved, for instance, by value iteration. Figure 4.3 shows the change
50
in the value function over successive
40
Final 30
sweeps of value iteration, and thepolicy(stake) 20
final policy found, for the case of10
ph = 0.4. This policy is optimal, but
1
1
25
50
75
99
not unique. In fact, there is a whole
Capitalfamily of optimal policies, all corresponding to ties for the argmax action selection with respect to the op- Figure 4.3: The solution to the gambler’s problem for
p = 0.4. The upper graph shows the value function
timal value function. Can you guess hfound by successive sweeps of value iteration. The
what the entire family looks like?
lower graph shows the final policy.
Exercise 4.8 Why does the optimal
policy for the gambler’s problem have such a curious form? In particular, for capital of 50
it bets it all on one flip, but for capital of 51 it does not. Why is this a good policy? ⇤Exercise 4.9 (programming) Implement value iteration for the gambler’s problem andsolve it for ph = 0.25 and ph = 0.55. In programming, you may find it convenient tointroduce two dummy states corresponding to termination with capital of 0 and 100,
giving them values of 0 and 1 respectively. Show your results graphically, as in Figure 4.3.
Are your results stable as ✓ ! 0?
⇤
Exercise 4.10 What is the analog of the value iteration update (4.10) for action values,
qk+1 (s, a)?
⇤4.5. Asynchronous Dynamic Programming

4.5

85

Asynchronous Dynamic Programming

A major drawback to the DP methods that we have discussed so far is that they involve
operations over the entire state set of the MDP, that is, they require sweeps of the state
set. If the state set is very large, then even a single sweep can be prohibitively expensive.
For example, the game of backgammon has over 1020 states. Even if we could performthe value iteration update on a million states per second, it would take over a thousand
years to complete a single sweep.
Asynchronous DP algorithms are in-place iterative DP algorithms that are not organizedin terms of systematic sweeps of the state set. These algorithms update the values ofstates in any order whatsoever, using whatever values of other states happen to beavailable. The values of some states may be updated several times before the values ofothers are updated once. To converge correctly, however, an asynchronous algorithm
must continue to update the values of all the states: it can’t ignore any state after somepoint in the computation. Asynchronous DP algorithms allow great flexibility in selecting
states to update.
For example, one version of asynchronous value iteration updates the value, in place, of
only one state, sk , on each step, k, using the value iteration update (4.10). If 0  < 1,asymptotic convergence to v⇤ is guaranteed given only that all states occur in the sequence{sk } an infinite number of times (the sequence could even be random).1 Similarly, it ispossible to intermix policy evaluation and value iteration updates to produce a kind of
asynchronous truncated policy iteration. Although the details of this and other moreunusual DP algorithms are beyond the scope of this book, it is clear that a few di↵erent
updates form building blocks that can be used flexibly in a wide variety of sweepless DPalgorithms.
Of course, avoiding sweeps does not necessarily mean that we can get away with lesscomputation. It just means that an algorithm does not need to get locked into any
hopelessly long sweep before it can make progress improving a policy. We can try totake advantage of this flexibility by selecting the states to which we apply updates so
as to improve the algorithm’s rate of progress. We can try to order the updates to letvalue information propagate from state to state in an efficient way. Some states may notneed their values updated as often as others. We might even try to skip updating somestates entirely if they are not relevant to optimal behavior. Some ideas for doing this are
discussed in Chapter 8.
Asynchronous algorithms also make it easier to intermix computation with real-time
interaction. To solve a given MDP, we can run an iterative DP algorithm at the sametime that an agent is actually experiencing the MDP. The agent’s experience can be used
to determine the states to which the DP algorithm applies its updates. At the same time,
the latest value and policy information from the DP algorithm can guide the agent’s
decision making. For example, we can apply updates to states as the agent visits them.
This makes it possible to focus the DP algorithm’s updates onto parts of the state set

1 In the undiscounted episodic case, it is possible that there are some orderings of updates that do not
result in convergence, but it is relatively easy to avoid these.

86

Chapter 4: Dynamic Programmingthat are most relevant to the agent. This kind of focusing is a repeated theme inreinforcement learning.

4.6

Generalized Policy Iteration

Policy iteration consists of two simultaneous, interacting processes, one making the valuefunction consistent with the current policy (policy evaluation), and the other making
the policy greedy with respect to the current value function (policy improvement). In
policy iteration, these two processes alternate, each completing before the other begins,
but this is not really necessary. In value iteration, for example, only a single iterationof policy evaluation is performed in between each policy improvement. In asynchronous
DP methods, the evaluation and improvement processes are interleaved at an evenfiner grain. In some cases a single state is updated in one process before returning
to the other. As long as both processes continue to update all states, the ultimate
result is typically the same—convergence to the optimal value function and an optimal
policy.
We use the term generalized policy iteration (GPI) to referto the general idea of letting policy-evaluation and policyevaluation
improvement processes interact, independent of the granularityVv⇡
and other details of the two processes. Almost all reinforcement
learning methods are well described as GPI. That is, all have ⇡Videntifiable policies and value functions, with the policy always⇡
greedy(V )being improved with respect to the value function and the valuefunction always being driven toward the value function for the
improvement
policy, as suggested by the diagram to the right. If both theevaluation process and the improvement process stabilize, thatis, no longer produce changes, then the value function and policy
must be optimal. The value function stabilizes only when it
is consistent with the current policy, and the policy stabilizes ⇡⇤v⇤
only when it is greedy with respect to the current value function.
Thus, both processes stabilize only when a policy has been found that is greedy withrespect to its own evaluation function. This implies that the Bellman optimality equation(4.1) holds, and thus that the policy and the value function are optimal.
The evaluation and improvement processes in GPI can be viewed as both competingand cooperating. They compete in the sense that they pull in opposing directions. Making
the policy greedy with respect to the value function typically makes the value function
incorrect for the changed policy, and making the value function consistent with the policy
typically causes that policy no longer to be greedy. In the long run, however, thesetwo processes interact to find a single joint solution: the optimal value function and an
optimal policy.
One might also think of the interaction between the evaluation and improvement
processes in GPI in terms of two constraints or goals—for example, as two lines in4.7. Efficiency of Dynamic Programming87

a two-dimensional space as suggested by the diagram to the right. Although the real geometry isv=v⇡
much more complicated than this, the diagram suggests what happens in the real case. Each processv, ⇡drives the value function or policy toward one ofv⇤ , ⇡⇤
the lines representing a solution to one of the twoy( v )
goals. The goals interact because the two lines aregreed⇡=
not orthogonal. Driving directly toward one goalcauses some movement away from the other goal.
Inevitably, however, the joint process is brought closer to the overall goal of optimality.
The arrows in this diagram correspond to the behavior of policy iteration in that eachtakes the system all the way to achieving one of the two goals completely. In GPI
one could also take smaller, incomplete steps toward each goal. In either case, the twoprocesses together achieve the overall goal of optimality even though neither is attempting
to achieve it directly.

4.7

Efficiency of Dynamic Programming

DP may not be practical for very large problems, but compared with other methods
for solving MDPs, DP methods are actually quite efficient. If we ignore a few technicaldetails, then, in the worst case, the time that DP methods take to find an optimal policy
is polynomial in the number of states and actions. If n and k denote the number of states
and actions, this means that a DP method takes a number of computational operations
that is less than some polynomial function of n and k. A DP method is guaranteed tofind an optimal policy in polynomial time even though the total number of (deterministic)
policies is k n . In this sense, DP is exponentially faster than any direct search in policy
space could be, because direct search would have to exhaustively examine each policy
to provide the same guarantee. Linear programming methods can also be used to solve
MDPs, and in some cases their worst-case convergence guarantees are better than thoseof DP methods. But linear programming methods become impractical at a much smaller
number of states than do DP methods (by a factor of about 100). For the largest problems,
only DP methods are feasible.
DP is sometimes thought to be of limited applicability because of the curse of dimensionality, the fact that the number of states often grows exponentially with the numberof state variables. Large state sets do create difficulties, but these are inherent difficultiesof the problem, not of DP as a solution method. In fact, DP is comparatively better
suited to handling large state spaces than competing methods such as direct search andlinear programming.
In practice, DP methods can be used with today’s computers to solve MDPs withmillions of states. Both policy iteration and value iteration are widely used, and it is notclear which, if either, is better in general. In practice, these methods usually converge
much faster than their theoretical worst-case run times, particularly if they are started
with good initial value functions or policies.

88

Chapter 4: Dynamic Programming

On problems with large state spaces, asynchronous DP methods are often preferred. To
complete even one sweep of a synchronous method requires computation and memory forevery state. For some problems, even this much memory and computation is impractical,
yet the problem is still potentially solvable because relatively few states occur along
optimal solution trajectories. Asynchronous methods and other variations of GPI can beapplied in such cases and may find good or optimal policies much faster than synchronous
methods can.

4.8

Summary

In this chapter we have become familiar with the basic ideas and algorithms of dynamic
programming as they relate to solving finite MDPs. Policy evaluation refers to the (typically) iterative computation of the value function for a given policy. Policy improvement
refers to the computation of an improved policy given the value function for that policy.
Putting these two computations together, we obtain policy iteration and value iteration,
the two most popular DP methods. Either of these can be used to reliably compute
optimal policies and value functions for finite MDPs given complete knowledge of the
MDP.
Classical DP methods operate in sweeps through the state set, performing an expected
update operation on each state. Each such operation updates the value of one statebased on the values of all possible successor states and their probabilities of occurring.
Expected updates are closely related to Bellman equations: they are little more thanthese equations turned into assignment statements. When the updates no longer result inany changes in value, convergence has occurred to values that satisfy the corresponding
Bellman equation. Just as there are four primary value functions (v⇡ , v⇤ , q⇡ , and q⇤ ),there are four corresponding Bellman equations and four corresponding expected updates.
An intuitive view of the operation of DP updates is given by their backup diagrams.
Insight into DP methods and, in fact, into almost all reinforcement learning methods,
can be gained by viewing them as generalized policy iteration (GPI). GPI is the general ideaof two interacting processes revolving around an approximate policy and an approximate
value function. One process takes the policy as given and performs some form of policy
evaluation, changing the value function to be more like the true value function for thepolicy. The other process takes the value function as given and performs some formof policy improvement, changing the policy to make it better, assuming that the valuefunction is its value function. Although each process changes the basis for the other,overall they work together to find a joint solution: a policy and value function that are
unchanged by either process and, consequently, are optimal. In some cases, GPI can beproved to converge, most notably for the classical DP methods that we have presented inthis chapter. In other cases convergence has not been proved, but still the idea of GPI
improves our understanding of the methods.
It is not necessary to perform DP methods in complete sweeps through the state
set. Asynchronous DP methods are in-place iterative methods that update states in anarbitrary order, perhaps stochastically determined and using out-of-date information.
Many of these methods can be viewed as fine-grained forms of GPI.

4.8. Summary

89

Finally, we note one last special property of DP methods. All of them update estimatesof the values of states based on estimates of the values of successor states. That is, they
update estimates on the basis of other estimates. We call this general idea bootstrapping.
Many reinforcement learning methods perform bootstrapping, even those that do not
require, as DP requires, a complete and accurate model of the environment. In the next
chapter we explore reinforcement learning methods that do not require a model and do
not bootstrap. In the chapter after that we explore methods that do not require a modelbut do bootstrap. These key features and properties are separable, yet can be mixed ininteresting combinations.

Bibliographical and Historical Remarks
The term “dynamic programming” is due to Bellman (1957a), who showed how thesemethods could be applied to a wide range of problems. Extensive treatments of DP canbe found in many texts, including Bertsekas (2005, 2012), Bertsekas and Tsitsiklis (1996),
Dreyfus and Law (1977), Ross (1983), White (1969), and Whittle (1982, 1983). Ourinterest in DP is restricted to its use in solving MDPs, but DP also applies to other typesof problems. Kumar and Kanal (1988) provide a more general look at DP.
To the best of our knowledge, the first connection between DP and reinforcement
learning was made by Minsky (1961) in commenting on Samuel’s checkers player. Ina footnote, Minsky mentioned that it is possible to apply DP to problems in which
Samuel’s backing-up process can be handled in closed analytic form. This remark may
have misled artificial intelligence researchers into believing that DP was restricted toanalytically tractable problems and therefore largely irrelevant to artificial intelligence.
Andreae (1969) mentioned DP in the context of reinforcement learning. Werbos (1977)
suggested an approach to approximating DP called “heuristic dynamic programming”
that emphasizes gradient-descent methods for continuous-state problems (Werbos, 1982,
1987, 1988, 1989, 1992). These methods are closely related to the reinforcement learningalgorithms that we discuss in this book. Watkins (1989) was explicit in connecting
reinforcement learning to DP, characterizing a class of reinforcement learning methods as“incremental dynamic programming.”
4.1–4 These sections describe well-established DP algorithms that are covered in any of
the general DP references cited above. The policy improvement theorem and thepolicy iteration algorithm are due to Bellman (1957a) and Howard (1960). Ourpresentation was influenced by the local view of policy improvement taken by
Watkins (1989). Our discussion of value iteration as a form of truncated policy
iteration is based on the approach of Puterman and Shin (1978), who presented aclass of algorithms called modified policy iteration, which includes policy iterationand value iteration as special cases. An analysis showing how value iteration canbe made to find an optimal policy in finite time is given by Bertsekas (1987).
Iterative policy evaluation is an example of a classical successive approximation
algorithm for solving a system of linear equations. The version of the algorithm

90

Chapter 4: Dynamic Programmingthat uses two arrays, one holding the old values while the other is updated, isoften called a Jacobi-style algorithm, after Jacobi’s classical use of this method.
It is also sometimes called a synchronous algorithm because the e↵ect is as if all
the values are updated at the same time. The second array is needed to simulate
this parallel computation sequentially. The in-place version of the algorithm
is often called a Gauss–Seidel-style algorithm after the classical Gauss–Seidel
algorithm for solving systems of linear equations. In addition to iterative policy
evaluation, other DP algorithms can be implemented in these di↵erent versions.
Bertsekas and Tsitsiklis (1989) provide excellent coverage of these variations andtheir performance di↵erences.
4.5

Asynchronous DP algorithms are due to Bertsekas (1982, 1983), who also called
them distributed DP algorithms. The original motivation for asynchronous
DP was its implementation on a multiprocessor system with communication
delays between processors and no global synchronizing clock. These algorithms
are extensively discussed by Bertsekas and Tsitsiklis (1989). Jacobi-style andGauss–Seidel-style DP algorithms are special cases of the asynchronous version.
Williams and Baird (1990) presented DP algorithms that are asynchronous at afiner grain than the ones we have discussed: the update operations themselves
are broken into steps that can be performed asynchronously.

4.7This section, written with the help of Michael Littman, is based on Littman,
Dean, and Kaelbling (1995). The phrase “curse of dimensionality” is due to
Bellman (1957a).
Foundational work on the linear programming approach to reinforcement learning
was done by Daniela de Farias (de Farias, 2002; de Farias and Van Roy, 2003).

Chapter 5Monte Carlo Methods
In this chapter we consider our first learning methods for estimating value functions anddiscovering optimal policies. Unlike the previous chapter, here we do not assume complete
knowledge of the environment. Monte Carlo methods require only experience—sample
sequences of states, actions, and rewards from actual or simulated interaction with anenvironment. Learning from actual experience is striking because it requires no priorknowledge of the environment’s dynamics, yet can still attain optimal behavior. Learningfrom simulated experience is also powerful. Although a model is required, the model need
only generate sample transitions, not the complete probability distributions of all possible
transitions that is required for dynamic programming (DP). In surprisingly many cases it
is easy to generate experience sampled according to the desired probability distributions,but infeasible to obtain the distributions in explicit form.
Monte Carlo methods are ways of solving the reinforcement learning problem based on
averaging sample returns. To ensure that well-defined returns are available, here we define
Monte Carlo methods only for episodic tasks. That is, we assume experience is divided
into episodes, and that all episodes eventually terminate no matter what actions are
selected. Only on the completion of an episode are value estimates and policies changed.
Monte Carlo methods can thus be incremental in an episode-by-episode sense, but not ina step-by-step (online) sense. The term “Monte Carlo” is often used more broadly forany estimation method whose operation involves a significant random component. Here
we use it specifically for methods based on averaging complete returns (as opposed tomethods that learn from partial returns, considered in the next chapter).
Monte Carlo methods sample and average returns for each state–action pair much likethe bandit methods we explored in Chapter 2 sample and average rewards for each action.
The main di↵erence is that now there are multiple states, each acting like a di↵erent
bandit problem (like an associative-search or contextual bandit) and the di↵erent bandit
problems are interrelated. That is, the return after taking an action in one state depends
on the actions taken in later states in the same episode. Because all the action selections
are undergoing learning, the problem becomes nonstationary from the point of view ofthe earlier state.

92

Chapter 5: Monte Carlo Methods

To handle the nonstationarity, we adapt the idea of general policy iteration (GPI)developed in Chapter 4 for DP. Whereas there we computed value functions from knowledgeof the MDP, here we learn value functions from sample returns with the MDP. The valuefunctions and corresponding policies still interact to attain optimality in essentially thesame way (GPI). As in the DP chapter, first we consider the prediction problem (thecomputation of v⇡ and q⇡ for a fixed arbitrary policy ⇡) then policy improvement, and,
finally, the control problem and its solution by GPI. Each of these ideas taken from DP
is extended to the Monte Carlo case in which only sample experience is available.

5.1

Monte Carlo Prediction

We begin by considering Monte Carlo methods for learning the state-value function for agiven policy. Recall that the value of a state is the expected return—expected cumulative
future discounted reward—starting from that state. An obvious way to estimate it fromexperience, then, is simply to average the returns observed after visits to that state. As
more returns are observed, the average should converge to the expected value. This idea
underlies all Monte Carlo methods.
In particular, suppose we wish to estimate v⇡ (s), the value of a state s under policy ⇡,given a set of episodes obtained by following ⇡ and passing through s. Each occurrenceof state s in an episode is called a visit to s. Of course, s may be visited multiple times
in the same episode; let us call the first time it is visited in an episode the first visit
to s. The first-visit MC method estimates v⇡ (s) as the average of the returns following
first visits to s, whereas the every-visit MC method averages the returns following all
visits to s. These two Monte Carlo (MC) methods are very similar but have slightlydi↵erent theoretical properties. First-visit MC has been most widely studied, dating backto the 1940s, and is the one we focus on in this chapter. Every-visit MC extends morenaturally to function approximation and eligibility traces, as discussed in Chapters 9 and12. First-visit MC is shown in procedural form in the box. Every-visit MC would be thesame except without the check for St having occurred earlier in the episode.
First-visit MC prediction, for estimating V ⇡ v⇡
Input: a policy ⇡ to be evaluatedInitialize:
V (s) 2 R, arbitrarily, for all s 2 S
Returns(s)an empty list, for all s 2 S
Loop forever (for each episode):
Generate an episode following ⇡: S0 , A0 , R1 , S1 , A1 , R2 , . . . , STG
0Loop for each step of episode, t = T 1, T 2, . . . , 0:G
G + Rt+1
Unless St appears in S0 , S1 , . . . , St 1 :Append G to Returns(St )V (St )
average(Returns(St ))1 , AT

1 , RT5.1. Monte Carlo Prediction

93

Both first-visit MC and every-visit MC converge to v⇡ (s) as the number of visits (orfirst visits) to s goes to infinity. This is easy to see for the case of first-visit MC. In
this case each return is an independent, identically distributed estimate of v⇡ (s) withfinite variance. By the law of large numbers the sequence of averages of these estimatesconverges to their expected value. Each p
average is itself an unbiased estimate, and thestandard deviation of its error falls as 1/ n, where n is the number of returns averaged.
Every-visit MC is less straightforward, but its estimates also converge quadratically tov⇡ (s) (Singh and Sutton, 1996).
The use of Monte Carlo methods is best illustrated through an example.
Example 5.1: Blackjack The object of the popular casino card game of blackjack is toobtain cards the sum of whose numerical values is as great as possible without exceeding
21. All face cards count as 10, and an ace can count either as 1 or as 11. We consider
the version in which each player competes independently against the dealer. The game
begins with two cards dealt to both dealer and player. One of the dealer’s cards is faceup and the other is face down. If the player has 21 immediately (an ace and a 10-card),
it is called a natural. He then wins unless the dealer also has a natural, in which case thegame is a draw. If the player does not have a natural, then he can request additionalcards, one by one (hits), until he either stops (sticks) or exceeds 21 (goes bust). If he goesbust, he loses; if he sticks, then it becomes the dealer’s turn. The dealer hits or sticks
according to a fixed strategy without choice: he sticks on any sum of 17 or greater, andhits otherwise. If the dealer goes bust, then the player wins; otherwise, the outcome—win,lose, or draw—is determined by whose final sum is closer to 21.
Playing blackjack is naturally formulated as an episodic finite MDP. Each game ofblackjack is an episode. Rewards of +1, 1, and 0 are given for winning, losing, anddrawing, respectively. All rewards within a game are zero, and we do not discount ( = 1);
therefore these terminal rewards are also the returns. The player’s actions are to hit or
to stick. The states depend on the player’s cards and the dealer’s showing card. We
assume that cards are dealt from an infinite deck (i.e., with replacement) so that there isno advantage to keeping track of the cards already dealt. If the player holds an ace thathe could count as 11 without going bust, then the ace is said to be usable. In this case
it is always counted as 11 because counting it as 1 would make the sum 11 or less, inwhich case there is no decision to be made because, obviously, the player should always
hit. Thus, the player makes decisions on the basis of three variables: his current sum(12–21), the dealer’s one showing card (ace–10), and whether or not he holds a usable
ace. This makes for a total of 200 states.
Consider the policy that sticks if the player’s sum is 20 or 21, and otherwise hits. To
find the state-value function for this policy by a Monte Carlo approach, one simulates
many blackjack games using the policy and averages the returns following each state.
In this way, we obtained the estimates of the state-value function shown in Figure 5.1.
The estimates for states with a usable ace are less certain and less regular because thesestates are less common. In any event, after 500,000 games the value function is very wellapproximated.

94

Chapter 5: Monte Carlo Methods

After 10,000 episodes

sum 2 1

!1

A

De

ale

r sh

yer

No
usable
ace

+1

ow

ing

10

12
Pla

Usableace

After 500,000 episodes

Figure 5.1: Approximate state-value functions for the blackjack policy that sticks only on 20
or 21, computed by Monte Carlo policy evaluation.

Exercise 5.1 Consider the diagrams on the right in Figure 5.1. Why does the estimatedvalue function jump up for the last two rows in the rear? Why does it drop o↵ for thewhole last row on the left? Why are the frontmost values higher in the upper diagrams
than in the lower?
⇤
Exercise 5.2 Suppose every-visit MC was used instead of first-visit MC on the blackjacktask. Would you expect the results to be very di↵erent? Why or why not?
⇤
Although we have complete knowledge of the environment in the blackjack task, it
would not be easy to apply DP methods to compute the value function. DP methods
require the distribution of next events—in particular, they require the environments
dynamics as given by the four-argument function p—and it is not easy to determine
this for blackjack. For example, suppose the player’s sum is 14 and he chooses to stick.
What is his probability of terminating with a reward of +1 as a function of the dealer’s
showing card? All of the probabilities must be computed before DP can be applied, andsuch computations are often complex and error-prone. In contrast, generating the sample
games required by Monte Carlo methods is easy. This is the case surprisingly often; theability of Monte Carlo methods to work with sample episodes alone can be a significant
advantage even when one has complete knowledge of the environment’s dynamics.
Can we generalize the idea of backup diagrams to Monte Carlo algorithms? The
general idea of a backup diagram is to show at the top the root node to be updated andto show below all the transitions and leaf nodes whose rewards and estimated values
contribute to the update. For Monte Carlo estimation of v⇡ , the root is a state node, andbelow it is the entire trajectory of transitions along a particular single episode, ending5.1. Monte Carlo Prediction

95

at the terminal state, as shown to the right. Whereas the DP diagram (page 59)
shows all possible transitions, the Monte Carlo diagram shows only those sampled
on the one episode. Whereas the DP diagram includes only one-step transitions,
the Monte Carlo diagram goes all the way to the end of the episode. Thesedi↵erences in the diagrams accurately reflect the fundamental di↵erences between
the algorithms.
An important fact about Monte Carlo methods is that the estimates for eachstate are independent. The estimate for one state does not build upon the estimate
of any other state, as is the case in DP. In other words, Monte Carlo methods do
not bootstrap as we defined it in the previous chapter.
In particular, note that the computational expense of estimating the value ofa single state is independent of the number of states. This can make Monte Carlo
methods particularly attractive when one requires the value of only one or a subsetof states. One can generate many sample episodes starting from the states of interest,
averaging returns from only these states, ignoring all others. This is a third advantage
Monte Carlo methods can have over DP methods (after the ability to learn from actual
experience and from simulated experience).
Example 5.2: Soap Bubble Suppose a wire
frame forming a closed loop is dunked in soapy
water to form a soap surface or bubble conforming at its edges to the wire frame. If the geometry of the wire frame is irregular but known,
how can you compute the shape of the surface?
The shape has the property that the total force
on each point exerted by neighboring points iszero (or else the shape would change). Thismeans that the surface’s height at any point is
the average of its heights at points in a small
A bubble on a wire loop.
circle around that point. In addition, the sur- From Hersh and Griego (1969). Reproduced withface must meet at its boundaries with the wire permission. ©1969 Scientific American, a division of Nature America, Inc. All rights reserved.
frame. The usual approach to problems of thiskind is to put a grid over the area covered by
the surface and solve for its height at the grid points by an iterative computation. Grid
points at the boundary are forced to the wire frame, and all others are adjusted toward
the average of the heights of their four nearest neighbors. This process then iterates, much
like DP’s iterative policy evaluation, and ultimately converges to a close approximation
to the desired surface.
This is similar to the kind of problem for which Monte Carlo methods were originallydesigned. Instead of the iterative computation described above, imagine standing on thesurface and taking a random walk, stepping randomly from grid point to neighboring
grid point, with equal probability, until you reach the boundary. It turns out that the
expected value of the height at the boundary is a close approximation to the height ofthe desired surface at the starting point (in fact, it is exactly the value computed by theiterative method described above). Thus, one can closely approximate the height of the96

Chapter 5: Monte Carlo Methods

surface at a point by simply averaging the boundary heights of many walks started at
the point. If one is interested in only the value at one point, or any fixed small set ofpoints, then this Monte Carlo method can be far more efficient than the iterative method
based on local consistency.

5.2

Monte Carlo Estimation of Action Values

If a model is not available, then it is particularly useful to estimate action values (thevalues of state–action pairs) rather than state values. With a model, state values alone aresufficient to determine a policy; one simply looks ahead one step and chooses whicheveraction leads to the best combination of reward and next state, as we did in the chapter on
DP. Without a model, however, state values alone are not sufficient. One must explicitly
estimate the value of each action in order for the values to be useful in suggesting a policy.
Thus, one of our primary goals for Monte Carlo methods is to estimate q⇤ . To achievethis, we first consider the policy evaluation problem for action values.
The policy evaluation problem for action values is to estimate q⇡ (s, a), the expected
return when starting in state s, taking action a, and thereafter following policy ⇡. The
Monte Carlo methods for this are essentially the same as just presented for state values,
except now we talk about visits to a state–action pair rather than to a state. A state–action pair s, a is said to be visited in an episode if ever the state s is visited and action
a is taken in it. The every-visit MC method estimates the value of a state–action pairas the average of the returns that have followed all the visits to it. The first-visit MC
method averages the returns following the first time in each episode that the state wasvisited and the action was selected. These methods converge quadratically, as before, tothe true expected values as the number of visits to each state–action pair approaches
infinity.
The only complication is that many state–action pairs may never be visited. If ⇡ isa deterministic policy, then in following ⇡ one will observe returns only for one of theactions from each state. With no returns to average, the Monte Carlo estimates of theother actions will not improve with experience. This is a serious problem because thepurpose of learning action values is to help in choosing among the actions available ineach state. To compare alternatives we need to estimate the value of all the actions fromeach state, not just the one we currently favor.
This is the general problem of maintaining exploration, as discussed in the context
of the k -armed bandit problem in Chapter 2. For policy evaluation to work for action
values, we must assure continual exploration. One way to do this is by specifying thatthe episodes start in a state–action pair, and that every pair has a nonzero probability of
being selected as the start. This guarantees that all state–action pairs will be visited an
infinite number of times in the limit of an infinite number of episodes. We call this theassumption of exploring starts.
The assumption of exploring starts is sometimes useful, but of course it cannot be
relied upon in general, particularly when learning directly from actual interaction with anenvironment. In that case the starting conditions are unlikely to be so helpful. The mostcommon alternative approach to assuring that all state–action pairs are encountered is5.3. Monte Carlo Control

97

to consider only policies that are stochastic with a nonzero probability of selecting all
actions in each state. We discuss two important variants of this approach in later sections.
For now, we retain the assumption of exploring starts and complete the presentation of afull Monte Carlo control method.
Exercise 5.3 What is the backup diagram for Monte Carlo estimation of q⇡ ?

5.3⇤

Monte Carlo Control

We are now ready to consider how Monte Carlo estimation can be used in control, thatis, to approximate optimal policies. The overall idea is to proceed according to the same
pattern as in the DP chapter, that is, according to the idea of generalized policy iteration(GPI). In GPI one maintains both an approximate policy andevaluation
an approximate value function. The value function is repeatedlyQq⇡altered to more closely approximate the value function for thecurrent policy, and the policy is repeatedly improved with respect
Q
to the current value function, as suggested by the diagram to ⇡the right. These two kinds of changes work against each other to⇡greedy(Q)some extent, as each creates a moving target for the other, but
improvement
together they cause both policy and value function to approach
optimality.
To begin, let us consider a Monte Carlo version of classical policy iteration. In
this method, we perform alternating complete steps of policy evaluation and policy
improvement, beginning with an arbitrary policy ⇡0 and ending with the optimal policy
and optimal action-value function:
E

I

E

I

E

I

E

⇡0 ! q ⇡ 0 ! ⇡1 ! q ⇡ 1 ! ⇡2 ! · · · ! ⇡⇤ ! q ⇤ ,EIwhere ! denotes a complete policy evaluation and ! denotes a complete policy
improvement. Policy evaluation is done exactly as described in the preceding section.
Many episodes are experienced, with the approximate action-value function approaching
the true function asymptotically. For the moment, let us assume that we do indeed
observe an infinite number of episodes and that, in addition, the episodes are generated
with exploring starts. Under these assumptions, the Monte Carlo methods will computeeach q⇡k exactly, for arbitrary ⇡k .
Policy improvement is done by making the policy greedy with respect to the current
value function. In this case we have an action-value function, and therefore no model isneeded to construct the greedy policy. For any action-value function q, the corresponding
greedy policy is the one that, for each s 2 S, deterministically chooses an action withmaximal action-value:.
⇡(s) = arg max q(s, a).
a

(5.1)

Policy improvement then can be done by constructing each ⇡k+1 as the greedy policy
with respect to q⇡k . The policy improvement theorem (Section 4.2) then applies to ⇡k

98

Chapter 5: Monte Carlo Methodsand ⇡k+1 because, for all s 2 S,q⇡k (s, ⇡k+1 (s))=

q⇡k (s, argmax q⇡k (s, a))=max q⇡k (s, a)aaq⇡k (s, ⇡k (s))
v⇡k (s).
As we discussed in the previous chapter, the theorem assures us that each ⇡k+1 is uniformly
better than ⇡k , or just as good as ⇡k , in which case they are both optimal policies. Thisin turn assures us that the overall process converges to the optimal policy and optimalvalue function. In this way Monte Carlo methods can be used to find optimal policies
given only sample episodes and no other knowledge of the environment’s dynamics.
We made two unlikely assumptions above in order to easily obtain this guarantee of
convergence for the Monte Carlo method. One was that the episodes have exploring
starts, and the other was that policy evaluation could be done with an infinite number of
episodes. To obtain a practical algorithm we will have to remove both assumptions. We
postpone consideration of the first assumption until later in this chapter.
For now we focus on the assumption that policy evaluation operates on an infinite
number of episodes. This assumption is relatively easy to remove. In fact, the same issue
arises even in classical DP methods such as iterative policy evaluation, which also converge
only asymptotically to the true value function. In both DP and Monte Carlo cases there
are two ways to solve the problem. One is to hold firm to the idea of approximating q⇡kin each policy evaluation. Measurements and assumptions are made to obtain bounds
on the magnitude and probability of error in the estimates, and then sufficient steps aretaken during each policy evaluation to assure that these bounds are sufficiently small.
This approach can probably be made completely satisfactory in the sense of guaranteeing
correct convergence up to some level of approximation. However, it is also likely to require
far too many episodes to be useful in practice on any but the smallest problems.
There is a second approach to avoiding the infinite number of episodes nominallyrequired for policy evaluation, in which we give up trying to complete policy evaluation
before returning to policy improvement. On each evaluation step we move the valuefunction toward q⇡k , but we do not expect to actually get close except over many steps.
We used this idea when we first introduced the idea of GPI in Section 4.6. One extreme
form of the idea is value iteration, in which only one iteration of iterative policy evaluationis performed between each step of policy improvement. The in-place version of value
iteration is even more extreme; there we alternate between improvement and evaluation
steps for single states.
For Monte Carlo policy iteration it is natural to alternate between evaluation andimprovement on an episode-by-episode basis. After each episode, the observed returns
are used for policy evaluation, and then the policy is improved at all the states visited in
the episode. A complete simple algorithm along these lines, which we call Monte Carlo
ES, for Monte Carlo with Exploring Starts, is given in pseudocode in the box on the nextpage.

5.3. Monte Carlo Control

99

Monte Carlo ES (Exploring Starts), for estimating ⇡ ⇡ ⇡⇤Initialize:⇡(s) 2 A(s) (arbitrarily), for all s 2 S
Q(s, a) 2 R (arbitrarily), for all s 2 S, a 2 A(s)Returns(s, a)empty list, for all s 2 S, a 2 A(s)Loop forever (for each episode):
Choose S0 2 S, A0 2 A(S0 ) randomly such that all pairs have probability > 0
Generate an episode from S0 , A0 , following ⇡: S0 , A0 , R1 , . . . , ST 1 , AT 1 , RTG
0Loop for each step of episode, t = T 1, T 2, . . . , 0:G
G + Rt+1
Unless the pair St , At appears in S0 , A0 , S1 , A1 . . . , St 1 , At 1 :Append G to Returns(St , At )Q(St , At )average(Returns(St , At ))⇡(St )
argmaxa Q(St , a)

Exercise 5.4 The pseudocode for Monte Carlo ES is inefficient because, for each state–action pair, it maintains a list of all returns and repeatedly calculates their mean. It would
be more efficient to use techniques similar to those explained in Section 2.4 to maintainjust the mean and a count (for each state–action pair) and update them incrementally.
Describe how the pseudocode would be altered to achieve this.
⇤In Monte Carlo ES, all the returns for each state–action pair are accumulated andaveraged, irrespective of what policy was in force when they were observed. It is easy
to see that Monte Carlo ES cannot converge to any suboptimal policy. If it did, thenthe value function would eventually converge to the value function for that policy, andthat in turn would cause the policy to change. Stability is achieved only when boththe policy and the value function are optimal. Convergence to this optimal fixed point
seems inevitable as the changes to the action-value function decrease over time, but has
not yet been formally proved. In our opinion, this is one of the most fundamental open
theoretical questions in reinforcement learning (for a partial solution, see Tsitsiklis, 2002).
Example 5.3: Solving Blackjack It is straightforward to apply Monte Carlo ES toblackjack. Because the episodes are all simulated games, it is easy to arrange for exploring
starts that include all possibilities. In this case one simply picks the dealer’s cards, theplayer’s sum, and whether or not the player has a usable ace, all at random with equal
probability. As the initial policy we use the policy evaluated in the previous blackjack
example, that which sticks only on 20 or 21. The initial action-value function can be zerofor all state–action pairs. Figure 5.2 shows the optimal policy for blackjack found byMonte Carlo ES. This policy is the same as the “basic” strategy of Thorp (1966) with thesole exception of the leftmost notch in the policy for a usable ace, which is not present
in Thorp’s strategy. We are uncertain of the reason for this discrepancy, but confident
that what is shown here is indeed the optimal policy for the version of blackjack we havedescribed.

Vv**

!*

21

Vv*** 10!*A 2 3 4 5 6 7 8 9 10DeDealer showing

No
Usableusable
ace
ace

STICK
STICK

HIT HITAA 22 33 44 55 66 77 88 9910
10

Dealer showing

um 2211

"1
A

ale

er s

21
21
20
20
19
19
18
18
17
17
16
16
15
15
14
14
13
13
12
12
11
11

+1

r sh

Vivng** 1100

ow

sum 2211

HIT HIT

21
20
19
18
17
16
15
14
13
12
11

Player sum

No
Usableusable
ace
ace

STICK
STICK

Player sum

A 2 3 4 5 6 7 8 9 10

12

!**

Chapter 5: Monte Carlo Methods

"1
A

Play

HIT

+1

12

Usable
ace

+1
"1
AA

DDee
aaleler rsshhooww
iningg

Play
er

100

21
20
19
18
17
16
15
14
13
12
11

10

12

STICK

er s

Play

Monte Carlo Control without Exploring
Startsin
Dealer showing

g

10

12

5.4

um 2 1

Player sum

21
20
Figure 5.2: The optimal policy
andstate-valuefunction for blackjack, found by Monte Carlo
19
STICK
No function shown was 18
ES. The state-value
computedfrom the action-value function found by
17
16
Monte Carlo usable
ES.
15
ace
A
HIT 14
13
De
12
ale
r sh
11
ow
A 2 3 4 5 6 7 8 9 10

How can we avoid the unlikely assumption of exploring starts? The only general way toensure that all actions are selected infinitely often is for the agent to continue to selectthem. There are two approaches to ensuring this, resulting in what we call on-policy
methods and o↵-policy methods. On-policy methods attempt to evaluate or improve thepolicy that is used to make decisions, whereas o↵-policy methods evaluate or improvea policy di↵erent from that used to generate the data. The Monte Carlo ES method
developed above is an example of an on-policy method. In this section we show how an
on-policy Monte Carlo control method can be designed that does not use the unrealistic
assumption of exploring starts. O↵-policy methods are considered in the next section.
In on-policy control methods the policy is generally soft, meaning that ⇡(a|s) > 0
for all s 2 S and all a 2 A(s), but gradually shifted closer and closer to a deterministic
optimal policy. Many of the methods discussed in Chapter 2 provide mechanisms forthis. The on-policy method we present in this section uses "-greedy policies, meaningthat most of the time they choose an action that has maximal estimated action value,
but with probability " they instead select an action at random. That is, all nongreedy"
actions are given the minimal probability of selection, |A(s)|, and the remaining bulk of"
the probability, 1 " + |A(s)| , is given to the greedy action. The "-greedy policies are"examples of "-soft policies, defined as policies for which ⇡(a|s) |A(s)|for all states andactions, for some " > 0. Among "-soft policies, "-greedy policies are in some sense thosethat are closest to greedy.

5.4. Monte Carlo Control without Exploring Starts

101

The overall idea of on-policy Monte Carlo control is still that of GPI. As in Monte
Carlo ES, we use first-visit MC methods to estimate the action-value function for thecurrent policy. Without the assumption of exploring starts, however, we cannot simply
improve the policy by making it greedy with respect to the current value function, becausethat would prevent further exploration of nongreedy actions. Fortunately, GPI does not
require that the policy be taken all the way to a greedy policy, only that it be moved
toward a greedy policy. In our on-policy method we will move it only to an "-greedy
policy. For any "-soft policy, ⇡, any "-greedy policy with respect to q⇡ is guaranteed tobe better than or equal to ⇡. The complete algorithm is given in the box below.
On-policy first-visit MC control (for "-soft policies), estimates ⇡ ⇡ ⇡⇤Algorithm parameter: small " > 0
Initialize:⇡
an arbitrary "-soft policy
Q(s, a) 2 R (arbitrarily), for all s 2 S, a 2 A(s)Returns(s, a)empty list, for all s 2 S, a 2 A(s)Repeat forever (for each episode):
Generate an episode following ⇡: S0 , A0 , R1 , . . . , ST 1 , AT 1 , RTG
0Loop for each step of episode, t = T 1, T 2, . . . , 0:G
G + Rt+1
Unless the pair St , At appears in S0 , A0 , S1 , A1 . . . , St 1 , At 1 :Append G to Returns(St , At )Q(St , At )average(Returns(St , At ))A⇤
argmaxa Q(St , a)(with ties broken arbitrarily)
For all a 2 A(St ):⇢1 " + "/|A(St )| if a = A⇤⇡(a|St )"/|A(St )|if a 6= A⇤
That any "-greedy policy with respect to q⇡ is an improvement over any "-soft policy⇡ is assured by the policy improvement theorem. Let ⇡ 0 be the "-greedy policy. The
conditions of the policy improvement theorem apply because for any s 2 S:
Xq⇡ (s, ⇡ 0 (s)) =⇡ 0 (a|s)q⇡ (s, a)
a

=

" X
q⇡ (s, a) + (1|A(s)| a" Xq⇡ (s, a) + (1|A(s)| a") max q⇡ (s, a)

(5.2)

a

")

X ⇡(a|s)a1"|A(s)|"q⇡ (s, a)102

Chapter 5: Monte Carlo Methods(the sum is a weighted average with nonnegative weights summing to 1, and as such it
must be less than or equal to the largest number averaged)
X
" X
" X
=
q⇡ (s, a)q⇡ (s, a) +⇡(a|s)q⇡ (s, a)|A(s)| a|A(s)| aa= v⇡ (s).

Thus, by the policy improvement theorem, ⇡ 0 ⇡ (i.e., v⇡0 (s) v⇡ (s), for all s 2 S). We
now prove that equality can hold only when both ⇡ 0 and ⇡ are optimal among the "-softpolicies, that is, when they are better than or equal to all other "-soft policies.
Consider a new environment that is just like the original environment, except with therequirement that policies be "-soft “moved inside” the environment. The new environment
has the same action and state set as the original and behaves as follows. If in state s
and taking action a, then with probability 1 " the new environment behaves exactly
like the old environment. With probability " it repicks the action at random, with equal
probabilities, and then behaves like the old environment with the new, random action.
The best one can do in this new environment with general policies is the same as thebest one could do in the original environment with "-soft policies. Let ve⇤ and qe⇤ denotethe optimal value functions for the new environment. Then a policy ⇡ is optimal among"-soft policies if and only if v⇡ = ve⇤ . We know that ve⇤ is the unique solution to the
Bellman optimality equation (3.19) with altered transition probabilities:
ih
i
Xh
X "
ve⇤ (s) = max(1 ")p(s0 , r |s, a) +p(s0 , r |s, a0 ) r + ve⇤ (s0 )a|A(s)|
s0 ,r
a0
h
i
X
= (1 ") max
p(s0 , r |s, a) r + ve⇤ (s0 )
a

+

s0 ,r

h
i
" XX
p(s0 , r |s, a) r + ve⇤ (s0 ) .
|A(s)| a 0s ,rWhen equality holds and the "-soft policy ⇡ is no longer improved, then we also know,
from (5.2), that" Xv⇡ (s) = (1 ") max q⇡ (s, a) +q⇡ (s, a)a|A(s)| ahiX= (1 ") max
p(s0 , r |s, a) r + v⇡ (s0 )
a

+

s0 ,r

h
i
" XX
p(s0 , r |s, a) r + v⇡ (s0 ) .
|A(s)| a 0s ,rHowever, this equation is the same as the previous one, except for the substitution of v⇡
for ve⇤ . Because ve⇤ is the unique solution, it must be that v⇡ = ve⇤ .
In essence, we have shown in the last few pages that policy iteration works for "-softpolicies. Using the natural notion of greedy policy for "-soft policies, one is assured of
improvement on every step, except when the best policy has been found among the "-softpolicies. This analysis is independent of how the action-value functions are determined5.5. O↵-policy Prediction via Importance Sampling103

at each stage, but it does assume that they are computed exactly. This brings us toroughly the same point as in the previous section. Now we only achieve the best policy
among the "-soft policies, but on the other hand, we have eliminated the assumption of
exploring starts.

5.5O↵-policy Prediction via Importance Sampling

All learning control methods face a dilemma: They seek to learn action values conditionalon subsequent optimal behavior, but they need to behave non-optimally in order toexplore all actions (to find the optimal actions). How can they learn about the optimal
policy while behaving according to an exploratory policy? The on-policy approach in thepreceding section is actually a compromise—it learns action values not for the optimal
policy, but for a near-optimal policy that still explores. A more straightforward approachis to use two policies, one that is learned about and that becomes the optimal policy, andone that is more exploratory and is used to generate behavior. The policy being learned
about is called the target policy, and the policy used to generate behavior is called the
behavior policy. In this case we say that learning is from data “o↵” the target policy, andthe overall process is termed o↵-policy learning.
Throughout the rest of this book we consider both on-policy and o↵-policy methods.
On-policy methods are generally simpler and are considered first. O↵-policy methods
require additional concepts and notation, and because the data is due to a di↵erent policy,o↵-policy methods are often of greater variance and are slower to converge. On the other
hand, o↵-policy methods are more powerful and general. They include on-policy methods
as the special case in which the target and behavior policies are the same. O↵-policy
methods also have a variety of additional uses in applications. For example, they canoften be applied to learn from data generated by a conventional non-learning controller,
or from a human expert. O↵-policy learning is also seen by some as key to learning
multi-step predictive models of the world’s dynamics (see Section 17.2; Sutton, 2009;
Sutton et al., 2011).
In this section we begin the study of o↵-policy methods by considering the prediction
problem, in which both target and behavior policies are fixed. That is, suppose we wish
to estimate v⇡ or q⇡ , but all we have are episodes following another policy b, whereb 6= ⇡. In this case, ⇡ is the target policy, b is the behavior policy, and both policies are
considered fixed and given.
In order to use episodes from b to estimate values for ⇡, we require that every action
taken under ⇡ is also taken, at least occasionally, under b. That is, we require that⇡(a|s) > 0 implies b(a|s) > 0. This is called the assumption of coverage. It follows
from coverage that b must be stochastic in states where it is not identical to ⇡. The
target policy ⇡, on the other hand, may be deterministic, and, in fact, this is a caseof particular interest in control applications. In control, the target policy is typicallythe deterministic greedy policy with respect to the current estimate of the action-valuefunction. This policy becomes a deterministic optimal policy while the behavior policy
remains stochastic and more exploratory, for example, an "-greedy policy. In this section,
however, we consider the prediction problem, in which ⇡ is unchanging and given.

104

Chapter 5: Monte Carlo Methods

Almost all o↵-policy methods utilize importance sampling, a general technique for
estimating expected values under one distribution given samples from another. We apply
importance sampling to o↵-policy learning by weighting returns according to the relative
probability of their trajectories occurring under the target and behavior policies, called
the importance-sampling ratio. Given a starting state St , the probability of the subsequent
state–action trajectory, At , St+1 , At+1 , . . . , ST , occurring under any policy ⇡ is
Pr{At , St+1 , At+1 , . . . , ST | St , At:T

1 ⇠ ⇡}= ⇡(At |St )p(St+1 |St , At )⇡(At+1 |St+1 ) · · · p(ST |ST
=

TY1
k=t

1 , AT

1)⇡(Ak |Sk )p(Sk+1 |Sk , Ak ),where p here is the state-transition probability function defined by (3.4). Thus, the relative
probability of the trajectory under the target and behavior policies (the importancesampling ratio) is
QT 1
TY1⇡(Ak |Sk )p(Sk+1 |Sk , Ak )⇡(Ak |Sk ).
⇢t:T 1 = Qk=t
=
.
(5.3)
T 1
b(Ak |Sk )k=t b(Ak |Sk )p(Sk+1 |Sk , Ak )k=tAlthough the trajectory probabilities depend on the MDP’s transition probabilities, which
are generally unknown, they appear identically in both the numerator and denominator,
and thus cancel. The importance sampling ratio ends up depending only on the twopolicies and the sequence, not on the MDP.
Recall that we wish to estimate the expected returns (values) under the target policy,
but all we have are returns Gt due to the behavior policy. These returns have the wrong
expectation E[Gt |St = s] = vb (s) and so cannot be averaged to obtain v⇡ . This is whereimportance sampling comes in. The ratio ⇢t:T 1 transforms the returns to have the rightexpected value:E[⇢t:T

1 Gt | St = s] = v⇡ (s).

(5.4)

Now we are ready to give a Monte Carlo algorithm that averages returns from a batchof observed episodes following policy b to estimate v⇡ (s). It is convenient here to number
time steps in a way that increases across episode boundaries. That is, if the first episodeof the batch ends in a terminal state at time 100, then the next episode begins at timet = 101. This enables us to use time-step numbers to refer to particular steps in particular
episodes. In particular, we can define the set of all time steps in which state s is visited,
denoted T(s). This is for an every-visit method; for a first-visit method, T(s) would only
include time steps that were first visits to s within their episodes. Also, let T (t) denotethe first time of termination following time t, and Gt denote the return after t up throughT (t). Then {Gt }t2T(s) are the returns that pertain to state s, and ⇢t:T (t) 1 t2T(s) are
the corresponding importance-sampling ratios. To estimate v⇡ (s), we simply scale thereturns by the ratios and average the results:P.
t2T(s) ⇢t:T (t) 1 Gt
V (s) =.
(5.5)
|T(s)|

5.5. O↵-policy Prediction via Importance Sampling

105

When importance sampling is done as a simple average in this way it is called ordinary
importance sampling.
An important alternative is weighted importance sampling, which uses a weighted
average, defined asP.
t2T(s) ⇢t:T (t) 1 Gt
V (s) = P,
(5.6)
t2T(s) ⇢t:T (t) 1

or zero if the denominator is zero. To understand these two varieties of importancesampling, consider the estimates of their first-visit methods after observing a single returnfrom state s. In the weighted-average estimate, the ratio ⇢t:T (t) 1 for the single returncancels in the numerator and denominator, so that the estimate is equal to the observed
return independent of the ratio (assuming the ratio is nonzero). Given that this return
was the only one observed, this is a reasonable estimate, but its expectation is vb (s) rather
than v⇡ (s), and in this statistical sense it is biased. In contrast, the first-visit versionof the ordinary importance-sampling estimator (5.5) is always v⇡ (s) in expectation (itis unbiased), but it can be extreme. Suppose the ratio were ten, indicating that the
trajectory observed is ten times as likely under the target policy as under the behavior
policy. In this case the ordinary importance-sampling estimate would be ten times theobserved return. That is, it would be quite far from the observed return even though theepisode’s trajectory is considered very representative of the target policy.
Formally, the di↵erence between the first-visit methods of the two kinds of importancesampling is expressed in their biases and variances. Ordinary importance sampling isunbiased whereas weighted importance sampling is biased (though the bias converges
asymptotically to zero). On the other hand, the variance of ordinary importance samplingis in general unbounded because the variance of the ratios can be unbounded, whereas in
the weighted estimator the largest weight on any single return is one. In fact, assuming
bounded returns, the variance of the weighted importance-sampling estimator converges
to zero even if the variance of the ratios themselves is infinite (Precup, Sutton, andDasgupta 2001). In practice, the weighted estimator usually has dramatically lower
variance and is strongly preferred. Nevertheless, we will not totally abandon ordinary
importance sampling as it is easier to extend to the approximate methods using function
approximation that we explore in the second part of this book.
The every-visit methods for ordinary and weighed importance sampling are both biased,though, again, the bias falls asymptotically to zero as the number of samples increases.
In practice, every-visit methods are often preferred because they remove the need to keep
track of which states have been visited and because they are much easier to extend to
approximations. A complete every-visit MC algorithm for o↵-policy policy evaluation
using weighted importance sampling is given in the next section on page 110.

Exercise 5.5 Consider an MDP with a single nonterminal state and a single actionthat transitions back to the nonterminal state with probability p and transitions to theterminal state with probability 1 p. Let the reward be +1 on all transitions, and let= 1. Suppose you observe one episode that lasts 10 steps, with a return of 10. What
are the first-visit and every-visit estimators of the value of the nonterminal state?
⇤

106

Chapter 5: Monte Carlo Methods

Example 5.4: O↵-policy Estimation of a Blackjack State Value We appliedboth ordinary and weighted importance-sampling methods to estimate the value of a single
blackjack state (Example 5.1) from o↵-policy data. Recall that one of the advantages ofMonte Carlo methods is that they can be used to evaluate a single state without forming
estimates for any other states. In this example, we evaluated the state in which the dealer
is showing a deuce, the sum of the player’s cards is 13, and the player has a usable ace (thatis, the player holds an ace and a deuce, or equivalently three aces). The data was generated
by starting in this state then choosing to hit or stick at random with equal probability(the behavior policy). The target policy was to stick only on a sum of 20 or 21, as in
Example 5.1. The value of this state under the target policy is approximately 0.27726(this was determined by separately generating one-hundred million episodes using thetarget policy and averaging their returns). Both o↵-policy methods closely approximatedthis value after 1000 o↵-policy episodes using the random policy. To make sure they did
this reliably, we performed 100 independent runs, each starting from estimates of zeroand learning for 10,000 episodes. Figure 5.3 shows the resultant learning curves—thesquared error of the estimates of each method as a function of number of episodes,
averaged over the 100 runs. The error approaches zero for both algorithms, but theweighted importance-sampling method has much lower error at the beginning, as is typicalin practice.
5

Ordinary
importancesampling

Meansquareerror(average over
100 runs)

Weighted importance sampling
0
0

10

100

1000

10,000

Episodes (log scale)Figure 5.3: Weighted importance sampling produces lower error estimates of the value of asingle blackjack state from o↵-policy episodes.

Example 5.5: Infinite Variance The estimates of ordinary importance sampling will
typically have infinite variance, and thus unsatisfactory convergence properties, whenever
the scaled returns have infinite variance—and this can easily happen in o↵-policy learningwhen trajectories contain loops. A simple example is shown inset in Figure 5.4. There is
only one nonterminal state s and two actions, right and left. The right action causes a
deterministic transition to termination, whereas the left action transitions, with probability
0.9, back to s or, with probability 0.1, on to termination. The rewards are +1 on thelatter transition and otherwise zero. Consider the target policy that always selects left.
All episodes under this policy consist of some number (possibly zero) of transitions back

5.5. O↵-policy Prediction via Importance Sampling107

to s followed by termination with a reward and return of +1. Thus the value of s under
the target policy is 1 ( = 1). Suppose we are estimating this value from o↵-policy datausing the behavior policy that selects right and left with equal probability.
R = +1⇡(left|s) = 1
left

0.1

s

0.9

right
R=0

b(left|s) =

1
2

R=0
2

Monte-Carloestimate ofv⇡ (s) withordinary
importancesampling(ten runs)

1

0
1

10

100

1000

10,000100,000

1,000,000

10,000,000

100,000,000

Episodes (log scale)Figure 5.4: Ordinary importance sampling produces surprisingly unstable estimates on the
one-state MDP shown inset (Example 5.5). The correct estimate here is 1 ( = 1), and, eventhough this is the expected value of a sample return (after importance sampling), the varianceof the samples is infinite, and the estimates do not converge to this value. These results are foro↵-policy first-visit MC.

The lower part of Figure 5.4 shows ten independent runs of the first-visit MC algorithm
using ordinary importance sampling. Even after millions of episodes, the estimates fail
to converge to the correct value of 1. In contrast, the weighted importance-samplingalgorithm would give an estimate of exactly 1 forever after the first episode that ended
with the left action. All returns not equal to 1 (that is, ending with the right action)
would be inconsistent with the target policy and thus would have a ⇢t:T (t) 1 of zero andcontribute neither to the numerator nor denominator of (5.6). The weighted importancesampling algorithm produces a weighted average of only the returns consistent with thetarget policy, and all of these would be exactly 1.
We can verify that the variance of the importance-sampling-scaled returns is infinite
in this example by a simple calculation. The variance of any random variable X is the
expected value of the deviation from its mean X̄, which can be writtenh.
Var[X] = E X

X̄

2

i

⇥
= E X2⇤⇥ ⇤2X X̄ + X̄ 2 = E X 2

X̄ 2 .

Thus, if the mean is finite, as it is in our case, the variance is infinite if and only if the
expectation of the square of the random variable is infinite. Thus, we need only show

108

Chapter 5: Monte Carlo Methods

that the expected square of the importance-sampling-scaled return is infinite:
2

E4

TY1

⇡(At |St )G0b(At |St )
t=0

!2 3

5.

To compute this expectation, we break it down into cases based on episode length andtermination. First note that, for any episode ending with the right action, the importancesampling ratio is zero, because the target policy would never take this action; these
episodes thus contribute nothing to the expectation (the quantity in parenthesis will bezero) and can be ignored. We need only consider episodes that involve some number(possibly zero) of left actions that transition back to the nonterminal state, followed by aleft action transitioning to termination. All of these episodes have a return of 1, so theG0 factor can be ignored. To get the expected square we need only consider each lengthof episode, multiplying the probability of the episode’s occurrence by the square of its
importance-sampling ratio, and add these up:

=

1
· 0.1
2

✓

1
0.5

◆2

(the length 1 episode)◆2
1 1
0.5 0.5
✓
◆2
1
1
1
1 1 1
+ · 0.9 · · 0.9 · · 0.1
22
2
0.5 0.5 0.5
11+ · 0.9 · · 0.1
2
2✓(the length 2 episode)(the length 3 episode)

+ ···
1
1
X
X
= 0.1
0.9k · 2k · 2 = 0.2
1.8k = 1.
k=0

k=0

Exercise 5.6 What is the equation analogous to (5.6) for action values Q(s, a) instead ofstate values V (s), again given returns generated using b?
⇤
Exercise 5.7 In learning curves such as those shown in Figure 5.3 error generally decreases
with training, as indeed happened for the ordinary importance-sampling method. But for
the weighted importance-sampling method error first increased and then decreased. Why
do you think this happened?
⇤
Exercise 5.8 The results with Example 5.5 and shown in Figure 5.4 used a first-visit MC
method. Suppose that instead an every-visit MC method was used on the same problem.
Would the variance of the estimator still be infinite? Why or why not?
⇤

5.7. O↵-policy Monte Carlo Control

5.6

109

Incremental Implementation

Monte Carlo prediction methods can be implemented incrementally, on an episode-byepisode basis, using extensions of the techniques described in Chapter 2 (Section 2.4).
Whereas in Chapter 2 we averaged rewards, in Monte Carlo methods we average returns.
In all other respects exactly the same methods as used in Chapter 2 can be used for onpolicy Monte Carlo methods. For o↵-policy Monte Carlo methods, we need to separately
consider those that use ordinary importance sampling and those that use weighted
importance sampling.
In ordinary importance sampling, the returns are scaled by the importance sampling
ratio ⇢t:T (t) 1 (5.3), then simply averaged, as in (5.5). For these methods we can again
use the incremental methods of Chapter 2, but using the scaled returns in place ofthe rewards of that chapter. This leaves the case of o↵-policy methods using weighted
importance sampling. Here we have to form a weighted average of the returns, and aslightly di↵erent incremental algorithm is required.
Suppose we have a sequence of returns G1 , G2 , . . . , Gn 1 , all starting in the same stateand each with a corresponding random weight Wi (e.g., Wi = ⇢ti :T (ti ) 1 ). We wish toform the estimate
.
Vn =

Pn 1

k=1 Wk Gk,Pn 1
k=1 Wk

n

2,

(5.7)

and keep it up-to-date as we obtain a single additional return Gn . In addition to keeping
track of Vn , we must maintain for each state the cumulative sum Cn of the weights given
to the first n returns. The update rule for Vn is
Wn h.
Vn+1 = Vn +
Gn
Cn

i
Vn ,

n

1,

(5.8)

and
.
Cn+1 = Cn + Wn+1 ,.
where C0 = 0 (and V1 is arbitrary and thus need not be specified). The box on thenext page contains a complete episode-by-episode incremental algorithm for Monte Carlo
policy evaluation. The algorithm is nominally for the o↵-policy case, using weighted
importance sampling, but applies as well to the on-policy case just by choosing thetarget and behavior policies as the same (in which case (⇡ = b), W is always 1). The
approximation Q converges to q⇡ (for all encountered state–action pairs) while actions
are selected according to a potentially di↵erent policy, b.
Exercise 5.9 Modify the algorithm for first-visit MC policy evaluation (Section 5.1) to
use the incremental implementation for sample averages described in Section 2.4.
⇤
Exercise 5.10 Derive the weighted-average update rule (5.8) from (5.7). Follow thepattern of the derivation of the unweighted rule (2.3).
⇤

110

Chapter 5: Monte Carlo Methods

O↵-policy MC prediction (policy evaluation) for estimating Q ⇡ q⇡
Input: an arbitrary target policy ⇡Initialize, for all s 2 S, a 2 A(s):Q(s, a) 2 R (arbitrarily)C(s, a)0Loop forever (for each episode):b
any policy with coverage of ⇡Generate an episode following b: S0 , A0 , R1 , . . . , ST 1 , AT 1 , RT
G0W
1Loop for each step of episode, t = T 1, T 2, . . . , 0, while W 6= 0:
G
G + Rt+1
C(St , At )C(St , At ) + W
Q(St , At )Q(St , At ) + C(SW[G Q(St , At )]t ,At )
W

5.7

t |St )W ⇡(Ab(At |St )O↵-policy Monte Carlo Control

We are now ready to present an example of the second class of learning control methods
we consider in this book: o↵-policy methods. Recall that the distinguishing feature of
on-policy methods is that they estimate the value of a policy while using it for control.
In o↵-policy methods these two functions are separated. The policy used to generate
behavior, called the behavior policy, may in fact be unrelated to the policy that isevaluated and improved, called the target policy. An advantage of this separation isthat the target policy may be deterministic (e.g., greedy), while the behavior policy cancontinue to sample all possible actions.
O↵-policy Monte Carlo control methods use one of the techniques presented in thepreceding two sections. They follow the behavior policy while learning about andimproving the target policy. These techniques require that the behavior policy has anonzero probability of selecting all actions that might be selected by the target policy(coverage). To explore all possibilities, we require that the behavior policy be soft (i.e.,that it select all actions in all states with nonzero probability).
The box on the next page shows an o↵-policy Monte Carlo control method, based on
GPI and weighted importance sampling, for estimating ⇡⇤ and q⇤ . The target policy⇡ ⇡ ⇡⇤ is the greedy policy with respect to Q, which is an estimate of q⇡ . The behavior
policy b can be anything, but in order to assure convergence of ⇡ to the optimal policy, an
infinite number of returns must be obtained for each pair of state and action. This can beassured by choosing b to be "-soft. The policy ⇡ converges to optimal at all encounteredstates even though actions are selected according to a di↵erent soft policy b, which may
change between or even within episodes.

5.7. O↵-policy Monte Carlo Control

111

O↵-policy MC control, for estimating ⇡ ⇡ ⇡⇤Initialize, for all s 2 S, a 2 A(s):Q(s, a) 2 R (arbitrarily)C(s, a)0⇡(s)
argmaxa Q(s, a) (with ties broken consistently)
Loop forever (for each episode):b
any soft policy
Generate an episode using b: S0 , A0 , R1 , . . . , ST 1 , AT 1 , RT
G0W
1Loop for each step of episode, t = T 1, T 2, . . . , 0:G
G + Rt+1
C(St , At )C(St , At ) + W
Q(St , At )Q(St , At ) + C(SW[G Q(St , At )]t ,At )⇡(St )
argmaxa Q(St , a) (with ties broken consistently)
If At 6= ⇡(St ) then exit inner Loop (proceed to next episode)W
W b(At1|St )A potential problem is that this method learns only from the tails of episodes, when
all of the remaining actions in the episode are greedy. If nongreedy actions are common,then learning will be slow, particularly for states appearing in the early portions oflong episodes. Potentially, this could greatly slow learning. There has been insufficientexperience with o↵-policy Monte Carlo methods to assess how serious this problem is. If
it is serious, the most important way to address it is probably by incorporating temporaldi↵erence learning, the algorithmic idea developed in the next chapter. Alternatively, if
is less than 1, then the idea developed in the next section may also help significantly.
Exercise 5.11 In the boxed algorithm for o↵-policy MC control, you may have beent |St )
expecting the W update to have involved the importance-sampling ratio ⇡(Ab(At |St ) , butinstead it involves b(At1|St ) . Why is this nevertheless correct?
⇤
Exercise 5.12: Racetrack (programming) Consider driving a race car around a turnlike those shown in Figure 5.5. You want to go as fast as possible, but not so fast as
to run o↵ the track. In our simplified racetrack, the car is at one of a discrete set of
grid positions, the cells in the diagram. The velocity is also discrete, a number of gridcells moved horizontally and vertically per time step. The actions are increments to thevelocity components. Each may be changed by +1, 1, or 0 in each step, for a total ofnine (3 ⇥ 3) actions. Both velocity components are restricted to be nonnegative and lessthan 5, and they cannot both be zero except at the starting line. Each episode begins
in one of the randomly selected start states with both velocity components zero andends when the car crosses the finish line. The rewards are 1 for each step until the carcrosses the finish line. If the car hits the track boundary, it is moved back to a random
position on the starting line, both velocity components are reduced to zero, and the

112

Chapter 5: Monte Carlo Methods

Finishline

Starting line

Finishline

Starting line

Figure 5.5: A couple of right turns for the racetrack task.

episode continues. Before updating the car’s location at each time step, check to see if
the projected path of the car intersects the track boundary. If it intersects the finish line,
the episode ends; if it intersects anywhere else, the car is considered to have hit the trackboundary and is sent back to the starting line. To make the task more challenging, withprobability 0.1 at each time step the velocity increments are both zero, independently of
the intended increments. Apply a Monte Carlo control method to this task to computethe optimal policy from each starting state. Exhibit several trajectories following the
optimal policy (but turn the noise o↵ for these trajectories).
⇤

5.8*Discounting-aware Importance Sampling

The o↵-policy methods that we have considered so far are based on forming importancesampling weights for returns considered as unitary wholes, without taking into accountthe returns’ internal structures as sums of discounted rewards. We now briefly consider
cutting-edge research ideas for using this structure to significantly reduce the variance ofo↵-policy estimators.
For example, consider the case where episodes are long and is significantly less than
1. For concreteness, say that episodes last 100 steps and that = 0. The return fromtime 0 will then be just G0 = R1 , but its importance sampling ratio will be a product of⇡(A99 |S99 )0 |S0 ) ⇡(A1 |S1 )100 factors, ⇡(Ab(A0 |S0 ) b(A1 |S1 ) · · · b(A99 |S99 ) . In ordinary importance sampling, the return

will be scaled by the entire product, but it is really only necessary to scale by the first⇡(A1 |S1 )⇡(A99 |S99 )0 |S0 )
factor, by ⇡(Ab(A0 |S0 ) . The other 99 factors b(A1 |S1 ) · · · b(A99 |S99 ) are irrelevant because

after the first reward the return has already been determined. These later factors are
all independent of the return and of expected value 1; they do not change the expected
update, but they add enormously to its variance. In some cases they could even make thevariance infinite. Let us now consider an idea for avoiding this large extraneous variance.

5.9. Per-decision Importance Sampling

113

The essence of the idea is to think of discounting as determining a probability oftermination or, equivalently, a degree of partial termination. For any 2 [0, 1), we canthink of the return G0 as partly terminating in one step, to the degree 1, producing
a return of just the first reward, R1 , and as partly terminating after two steps, to thedegree (1) , producing a return of R1 + R2 , and so on. The latter degree corresponds
to terminating on the second step, 1, and not having already terminated on thefirst step, . The degree of termination on the third step is thus (1) 2 , with the 2reflecting that termination did not occur on either of the first two steps. The partial
returns here are called flat partial returns:.
Ḡt:h = Rt+1 + Rt+2 + · · · + Rh ,0  t < h  T,
where “flat” denotes the absence of discounting, and “partial” denotes that these returns
do not extend all the way to termination but instead stop at h, called the horizon (and T
is the time of termination of the episode). The conventional full return Gt can be viewed
as a sum of flat partial returns as suggested above as follows:.
Gt = Rt+1 + Rt+2 + 2 Rt+3 + · · · + T t 1 RT
= (1

)Rt+1

+ (1

) (Rt+1 + Rt+2 )

+ (1
..
.

) 2 (Rt+1 + Rt+2 + Rt+3 )

+ (1

) T

+
= (1

T

t 1

)

t 2

(Rt+1 + Rt+2 + · · · + RT1)

(Rt+1 + Rt+2 + · · · + RT )

T
X1

h t 1

Ḡt:h +T

t 1

Ḡt:T .

h=t+1

Now we need to scale the flat partial returns by an importance sampling ratio thatis similarly truncated. As Ḡt:h only involves rewards up to a horizon h, we only need
the ratio of the probabilities up to h 1. We define an ordinary importance-sampling
estimator, analogous to (5.5), as⇣⌘P
PT (t) 1 h t 1
T (t) t 1
(1
)
⇢
Ḡ
+
⇢
Ḡ
t:h
1
t:h
t:T
(t)
1
t:T
(t)
t2T(s)
h=t+1.
V (s) =, (5.9)|T(s)|
and a weighted importance-sampling estimator, analogous to (5.6), as⇣⌘P
PT (t) 1) h=t+1 h t 1 ⇢t:h 1 Ḡt:h + T (t) t 1 ⇢t:T (t) 1 Ḡt:T (t)t2T(s) (1.
⇣⌘V (s) =. (5.10)P
PT (t) 1) h=t+1 h t 1 ⇢t:h 1 + T (t) t 1 ⇢t:T (t) 1
t2T(s) (1We call these two estimators discounting-aware importance sampling estimators. They
take into account the discount rate but have no e↵ect (are the same as the o↵-policy
estimators from Section 5.5) if = 1.

114

Chapter 5: Monte Carlo Methods

5.9*Per-decision Importance Sampling

There is one more way in which the structure of the return as a sum of rewards can betaken into account in o↵-policy importance sampling, a way that may be able to reduce
variance even in the absence of discounting (that is, even if = 1). In the o↵-policy
estimators (5.5) and (5.6), each term of the sum in the numerator is itself a sum:⇢t:T

1 Gt = ⇢t:T

= ⇢t:T1

Rt+1 + Rt+2 + · · · +1 Rt+1 +

⇢t:T

T

t 1

1 Rt+2 + · · · +

T

RT
t 1

⇢t:T

1 RT .

(5.11)The o↵-policy estimators rely on the expected values of these terms, which can be written
in a simpler way. Note that each sub-term of (5.11) is a product of a random reward anda random importance-sampling ratio. For example, the first sub-term can be written,
using (5.3), as⇢t:T

1 Rt+1 =⇡(At |St ) ⇡(At+1 |St+1 ) ⇡(At+2 |St+2 )⇡(AT 1 |ST 1 )···Rt+1 . (5.12)b(At |St ) b(At+1 |St+1 ) b(At+2 |St+2 )b(AT 1 |ST 1 )Of all these factors, one might suspect that only the first and the last (the reward)
are related; all the others are for events that occurred after the reward. Moreover, theexpected value of all these other factors is one:E⇡(Ak |Sk ) . X⇡(a|Sk ) X=b(a|Sk )=⇡(a|Sk ) = 1.
b(Ak |Sk )b(a|Sk )aa(5.13)

With a few more steps, one can show that, as suspected, all of these other factors haveno e↵ect in expectation, in other words, thatE[⇢t:T

1 Rt+1 ] = E[⇢t:t Rt+1 ] .

(5.14)

If we repeat this process for the kth sub-term of (5.11), we getE[⇢t:T

1 Rt+k ] = E[⇢t:t+k 1 Rt+k ] .

It follows then that the expectation of our original term (5.11) can be writtenE[⇢t:Th

1 Gt ] = E G̃t

i

,

where
G̃t = ⇢t:t Rt+1 + ⇢t:t+1 Rt+2 +2⇢t:t+2 Rt+3 + · · · +We call this idea per-decision importance sampling.

T

t 1

⇢t:T

1 RT .

5.10. Summary

115

It follows immediately that there is an alternate importance-sampling estimator, withthe same unbiased expectation (in the first-visit case) as the ordinary-importance-sampling
estimator (5.5), using G̃t :.
V (s) =P

t2T(s) G̃t|T(s)|,(5.15)

which we might expect to sometimes be of lower variance.
Is there a per-decision version of weighted importance sampling? This is less clear. So
far, all the estimators that have been proposed for this that we know of are not consistent(that is, they do not converge to the true value with infinite data).
⇤

Exercise 5.13 Show the steps to derive (5.14) from (5.12).

⇤⇤

Exercise 5.14 Modify the algorithm for o↵-policy Monte Carlo control (page 111) to use
the idea of the truncated weighted-average estimator (5.10). Note that you will first needto convert this equation to action values.
⇤

5.10

Summary

The Monte Carlo methods presented in this chapter learn value functions and optimal
policies from experience in the form of sample episodes. This gives them at least threekinds of advantages over DP methods. First, they can be used to learn optimal behavior
directly from interaction with the environment, with no model of the environment’s
dynamics. Second, they can be used with simulation or sample models. For surprisingly
many applications it is easy to simulate sample episodes even though it is difficult toconstruct the kind of explicit model of transition probabilities required by DP methods.
Third, it is easy and efficient to focus Monte Carlo methods on a small subset of the states.
A region of special interest can be accurately evaluated without going to the expense of
accurately evaluating the rest of the state set (we explore this further in Chapter 8).
A fourth advantage of Monte Carlo methods, which we discuss later in the book, isthat they may be less harmed by violations of the Markov property. This is because they
do not update their value estimates on the basis of the value estimates of successor states.
In other words, it is because they do not bootstrap.
In designing Monte Carlo control methods we have followed the overall schema ofgeneralized policy iteration (GPI) introduced in Chapter 4. GPI involves interacting
processes of policy evaluation and policy improvement. Monte Carlo methods provide analternative policy evaluation process. Rather than use a model to compute the value ofeach state, they simply average many returns that start in the state. Because a state’svalue is the expected return, this average can become a good approximation to thevalue. In control methods we are particularly interested in approximating action-valuefunctions, because these can be used to improve the policy without requiring a model ofthe environment’s transition dynamics. Monte Carlo methods intermix policy evaluationand policy improvement steps on an episode-by-episode basis, and can be incrementally
implemented on an episode-by-episode basis.

116

Chapter 5: Monte Carlo Methods

Maintaining sufficient exploration is an issue in Monte Carlo control methods. It is
not enough just to select the actions currently estimated to be best, because then no
returns will be obtained for alternative actions, and it may never be learned that they
are actually better. One approach is to ignore this problem by assuming that episodes
begin with state–action pairs randomly selected to cover all possibilities. Such exploring
starts can sometimes be arranged in applications with simulated episodes, but are unlikely
in learning from real experience. In on-policy methods, the agent commits to always
exploring and tries to find the best policy that still explores. In o↵-policy methods, theagent also explores, but learns a deterministic optimal policy that may be unrelated tothe policy followed.
O↵-policy prediction refers to learning the value function of a target policy from data
generated by a di↵erent behavior policy. Such learning methods are based on some formof importance sampling, that is, on weighting returns by the ratio of the probabilities oftaking the observed actions under the two policies, thereby transforming their expectations
from the behavior policy to the target policy. Ordinary importance sampling uses a
simple average of the weighted returns, whereas weighted importance sampling uses aweighted average. Ordinary importance sampling produces unbiased estimates, but haslarger, possibly infinite, variance, whereas weighted importance sampling always has
finite variance and is preferred in practice. Despite their conceptual simplicity, o↵-policy
Monte Carlo methods for both prediction and control remain unsettled and are a subjectof ongoing research.
The Monte Carlo methods treated in this chapter di↵er from the DP methods treatedin the previous chapter in two major ways. First, they operate on sample experience,and thus can be used for direct learning without a model. Second, they do not bootstrap.
That is, they do not update their value estimates on the basis of other value estimates.
These two di↵erences are not tightly linked, and can be separated. In the next chapter
we consider methods that learn from experience, like Monte Carlo methods, but alsobootstrap, like DP methods.

Bibliographical and Historical Remarks
The term “Monte Carlo” dates from the 1940s, when physicists at Los Alamos devised
games of chance that they could study to help understand complex physical phenomena
relating to the atom bomb. Coverage of Monte Carlo methods in this sense can be foundin several textbooks (e.g., Kalos and Whitlock, 1986; Rubinstein, 1981).
5.1–2 Singh and Sutton (1996) distinguished between every-visit and first-visit MC
methods and proved results relating these methods to reinforcement learningalgorithms. The blackjack example is based on an example used by Widrow,
Gupta, and Maitra (1973). The soap bubble example is a classical Dirichletproblem whose Monte Carlo solution was first proposed by Kakutani (1945; see
Hersh and Griego, 1969; Doyle and Snell, 1984).
Barto and Du↵ (1994) discussed policy evaluation in the context of classicalMonte Carlo algorithms for solving systems of linear equations. They used the5.10. Summary

117

analysis of Curtiss (1954) to point out the computational advantages of Monte
Carlo policy evaluation for large problems.
5.3–4 Monte Carlo ES was introduced in the 1998 edition of this book. That may havebeen the first explicit connection between Monte Carlo estimation and control
methods based on policy iteration. An early use of Monte Carlo methods toestimate action values in a reinforcement learning context was by Michie andChambers (1968). In pole balancing (page 56), they used averages of episodedurations to assess the worth (expected balancing “life”) of each possible actionin each state, and then used these assessments to control action selections. Theirmethod is similar in spirit to Monte Carlo ES with every-visit MC estimates.
Narendra and Wheeler (1986) studied a Monte Carlo method for ergodic finite
Markov chains that used the return accumulated between successive visits to thesame state as a reward for adjusting a learning automaton’s action probabilities.
5.5

Efficient o↵-policy learning has become recognized as an important challengethat arises in several fields. For example, it is closely related to the idea of“interventions” and “counterfactuals” in probabilistic graphical (Bayesian) models(e.g., Pearl, 1995; Balke and Pearl, 1994). O↵-policy methods using importancesampling have a long history and yet still are not well understood. Weighted
importance sampling, which is also sometimes called normalized importancesampling (e.g., Koller and Friedman, 2009), is discussed by Rubinstein (1981),
Hesterberg (1988), Shelton (2001), and Liu (2001) among others.
The target policy in o↵-policy learning is sometimes referred to in the literatureas the “estimation” policy, as it was in the first edition of this book.

5.7

The racetrack exercise is adapted from Barto, Bradtke, and Singh (1995), andfrom Gardner (1973).

5.8

Our treatment of the idea of discounting-aware importance sampling is based on
the analysis of Sutton, Mahmood, Precup, and van Hasselt (2014). It has been
worked out most fully to date by Mahmood (2017; Mahmood, van Hasselt, andSutton, 2014).

5.9

Per-decision importance sampling was introduced by Precup, Sutton, and Singh(2000). They also combined o↵-policy learning with temporal-di↵erence learning,
eligibility traces, and approximation methods, introducing subtle issues that we
consider in later chapters.

Exercise 5.15 Make new equations analogous to the importance-sampling Monte Carloestimates (5.5) and (5.6), but for action value estimates Q(s, a). You will need newnotation T(s, a) for the time steps on which the state–action pair s, a is visited on theepisode. Do these estimates involve more or less importance-sampling correction?

Chapter 6

Temporal-Di↵erence Learning
If one had to identify one idea as central and novel to reinforcement learning, it would
undoubtedly be temporal-di↵erence (TD) learning. TD learning is a combination ofMonte Carlo ideas and dynamic programming (DP) ideas. Like Monte Carlo methods,
TD methods can learn directly from raw experience without a model of the environment’s
dynamics. Like DP, TD methods update estimates based in part on other learned
estimates, without waiting for a final outcome (they bootstrap). The relationship between
TD, DP, and Monte Carlo methods is a recurring theme in the theory of reinforcement
learning; this chapter is the beginning of our exploration of it. Before we are done, wewill see that these ideas and methods blend into each other and can be combined in many
ways. In particular, in Chapter 7 we introduce n-step algorithms, which provide a bridgefrom TD to Monte Carlo methods, and in Chapter 12 we introduce the TD( ) algorithm,
which seamlessly unifies them.
As usual, we start by focusing on the policy evaluation or prediction problem, theproblem of estimating the value function v⇡ for a given policy ⇡. For the control problem(finding an optimal policy), DP, TD, and Monte Carlo methods all use some variation ofgeneralized policy iteration (GPI). The di↵erences in the methods are primarily di↵erencesin their approaches to the prediction problem.

6.1

TD Prediction

Both TD and Monte Carlo methods use experience to solve the prediction problem. Given
some experience following a policy ⇡, both methods update their estimate V of v⇡ forthe nonterminal states St occurring in that experience. Roughly speaking, Monte Carlo
methods wait until the return following the visit is known, then use that return as atarget for V (St ). A simple every-visit Monte Carlo method suitable for nonstationary
environments is
V (St )hV (St ) + ↵ Gti
V (St ) ,

(6.1)

120

Chapter 6: Temporal-Di↵erence Learningwhere Gt is the actual return following time t, and ↵ is a constant step-size parameter (c.f.,Equation 2.4). Let us call this method constant-↵ MC. Whereas Monte Carlo methods
must wait until the end of the episode to determine the increment to V (St ) (only then is
Gt known), TD methods need to wait only until the next time step. At time t + 1 they
immediately form a target and make a useful update using the observed reward Rt+1 andthe estimate V (St+1 ). The simplest TD method makes the updatehi
V (St )V (St ) + ↵ Rt+1 + V (St+1 ) V (St )(6.2)

immediately on transition to St+1 and receiving Rt+1 . In e↵ect, the target for the Monte
Carlo update is Gt , whereas the target for the TD update is Rt+1 + V (St+1 ). This TD
method is called TD(0), or one-step TD, because it is a special case of the TD( ) andn-step TD methods developed in Chapter 12 and Chapter 7. The box below specifies
TD(0) completely in procedural form.
Tabular TD(0) for estimating v⇡
Input: the policy ⇡ to be evaluatedAlgorithm parameter: step size ↵ 2 (0, 1]
Initialize V (s), for all s 2 S+ , arbitrarily except that V (terminal ) = 0
Loop for each episode:
Initialize S
Loop for each step of episode:Aaction given by ⇡ for S
Take action A, observe
R, S 0⇥
V (S)
V (S) + ↵ R + V (S 0 )0SSuntil S is terminal

V (S)

⇤

Because TD(0) bases its update in part on an existing estimate, we say that it is abootstrapping method, like DP. We know from Chapter 3 that.
v⇡ (s) = E⇡[Gt | St = s]

= E⇡[Rt+1 + Gt+1 | St = s]

= E⇡[Rt+1 + v⇡ (St+1 ) | St = s] .

(6.3)
(from (3.9))(6.4)

Roughly speaking, Monte Carlo methods use an estimate of (6.3) as a target, whereas
DP methods use an estimate of (6.4) as a target. The Monte Carlo target is an estimate
because the expected value in (6.3) is not known; a sample return is used in place of thereal expected return. The DP target is an estimate not because of the expected values,
which are assumed to be completely provided by a model of the environment, but becausev⇡ (St+1 ) is not known and the current estimate, V (St+1 ), is used instead. The TD target
is an estimate for both reasons: it samples the expected values in (6.4) and it uses thecurrent estimate V instead of the true v⇡ . Thus, TD methods combine the sampling of

6.1. TD Prediction

121

Monte Carlo with the bootstrapping of DP. As we shall see, with care and imagination
this can take us a long way toward obtaining the advantages of both Monte Carlo andDP methods.
Shown to the right is the backup diagram for tabular TD(0). The value
estimate for the state node at the top of the backup diagram is updated on
the basis of the one sample transition from it to the immediately following
state. We refer to TD and Monte Carlo updates as sample updates because
they involve looking ahead to a sample successor state (or state–action pair),using the value of the successor and the reward along the way to compute abacked-up value, and then updating the value of the original state (or state– TD(0)action pair) accordingly. Sample updates di↵er from the expected updatesof DP methods in that they are based on a single sample successor rather than on acomplete distribution of all possible successors.
Finally, note that the quantity in brackets in the TD(0) update is a sort of error,
measuring the di↵erence between the estimated value of St and the better estimate
Rt+1 + V (St+1 ). This quantity, called the TD error, arises in various forms throughout
reinforcement learning:.

t = Rt+1 +V (St+1 )V (St ).

(6.5)

Notice that the TD error at each time is the error in the estimate made at that time.
Because the TD error depends on the next state and next reward, it is not actuallyavailable until one time step later. That is, t is the error in V (St ), available at timet + 1. Also note that if the array V does not change during the episode (as it does not inMonte Carlo methods), then the Monte Carlo error can be written as a sum of TD errors:
Gt

V (St ) = Rt+1 + Gt+1
=
=
=
=
=

t+
t+
t+
t+
T
X1

V (St ) + V (St+1 )Gt+1V (St+1 )

t+1 +

2

Gt+2

t+1 +

2

t+1 +

2

t+2 + · · · +

k t

k.

V (St+1 )(from (3.9))

V (St+2 )t+2 + · · · +

T

t 1

T

t 1

T
T

1+

T

t

GT

1+

T

t

0

V (ST )
0
(6.6)

k=t

This identity is not exact if V is updated during the episode (as it is in TD(0)), but if thestep size is small then it may still hold approximately. Generalizations of this identity
play an important role in the theory and algorithms of temporal-di↵erence learning.
Exercise 6.1 If V changes during the episode, then (6.6) only holds approximately; what
would the di↵erence be between the two sides? Let Vt denote the array of state values
used at time t in the TD error (6.5) and in the TD update (6.2). Redo the derivation
above to determine the additional amount that must be added to the sum of TD errors
in order to equal the Monte Carlo error.
⇤

122

Chapter 6: Temporal-Di↵erence Learning

Example 6.1: Driving Home Each day as you drive home from work, you try to
predict how long it will take to get home. When you leave your office, you note the time,the day of week, the weather, and anything else that might be relevant. Say on thisFriday you are leaving at exactly 6 o’clock, and you estimate that it will take 30 minutes
to get home. As you reach your car it is 6:05, and you notice it is starting to rain. Traffic
is often slower in the rain, so you reestimate that it will take 35 minutes from then, or a
total of 40 minutes. Fifteen minutes later you have completed the highway portion ofyour journey in good time. As you exit onto a secondary road you cut your estimate of
total travel time to 35 minutes. Unfortunately, at this point you get stuck behind a slow
truck, and the road is too narrow to pass. You end up having to follow the truck until
you turn onto the side street where you live at 6:40. Three minutes later you are home.
The sequence of states, times, and predictions is thus as follows:Stateleaving office, friday at 6reach car, raining
exiting highway
2ndary road, behind truck
entering home street
arrive home

Elapsed Time(minutes)
0
5
20
30
40
43

Predicted
Time to Go
30
35
15
10
3
0

Predicted
Total Time
30
40
35
40
43
43

The rewards in this example are the elapsed times on each leg of the journey.1 We are
not discounting ( = 1), and thus the return for each state is the actual time to go fromthat state. The value of each state is the expected time to go. The second column of
numbers gives the current estimated value for each state encountered.
A simple way to view the operation of Monte Carlo methods is to plot the predicted
total time (the last column) over the sequence, as in Figure 6.1 (left). The red arrows
show the changes in predictions recommended by the constant-↵ MC method (6.1), for↵ = 1. These are exactly the errors between the estimated value (predicted time to go)in each state and the actual return (actual time to go). For example, when you exitedthe highway you thought it would take only 15 minutes more to get home, but in fact it
took 23 minutes. Equation 6.1 applies at this point and determines an increment in theestimate of time to go after exiting the highway. The error, Gt V (St ), at this time iseight minutes. Suppose the step-size parameter, ↵, is 1/2. Then the predicted time to go
after exiting the highway would be revised upward by four minutes as a result of thisexperience. This is probably too large a change in this case; the truck was probably justan unlucky break. In any event, the change can only be made o↵-line, that is, after you
have reached home. Only at this point do you know any of the actual returns.
Is it necessary to wait until the final outcome is known before learning can begin?
Suppose on another day you again estimate when leaving your office that it will take 30
minutes to drive home, but then you become stuck in a massive traffic jam. Twenty-fiveminutes after leaving the office you are still bumper-to-bumper on the highway. You now
1 If this were a control problem with the objective of minimizing travel time, then we would of course
make the rewards the negative of the elapsed time. But because we are concerned here only withprediction (policy evaluation), we can keep things simple by using positive numbers.

6.1. TD Prediction

45

123

45

actual outcome

40

actualoutcome40

Predicted
total
travel35
time

Predicted
total
travel
35
time

30

30
leaving reach exiting 2ndary home
car highway road street
officeSituation

arrivehomeleaving reach exiting 2ndary home
car highway road street
officearrivehome

Situation

Figure 6.1: Changes recommended in the driving home example by Monte Carlo methods (left)
and TD methods (right).

estimate that it will take another 25 minutes to get home, for a total of 50 minutes. As
you wait in traffic, you already know that your initial estimate of 30 minutes was too
optimistic. Must you wait until you get home before increasing your estimate for the
initial state? According to the Monte Carlo approach you must, because you don’t yet
know the true return.
According to a TD approach, on the other hand, you would learn immediately, shiftingyour initial estimate from 30 minutes toward 50. In fact, each estimate would be shifted
toward the estimate that immediately follows it. Returning to our first day of driving,
Figure 6.1 (right) shows the changes in the predictions recommended by the TD rule(6.2) (these are the changes made by the rule if ↵ = 1). Each error is proportional to thechange over time of the prediction, that is, to the temporal di↵erences in predictions.
Besides giving you something to do while waiting in traffic, there are several computational reasons why it is advantageous to learn based on your current predictions rather
than waiting until termination when you know the actual return. We briefly discuss someof these in the next section.
Exercise 6.2 This is an exercise to help develop your intuition about why TD methods
are often more efficient than Monte Carlo methods. Consider the driving home example
and how it is addressed by TD and Monte Carlo methods. Can you imagine a scenario
in which a TD update would be better on average than a Monte Carlo update? Givean example scenario—a description of past experience and a current state—in which
you would expect the TD update to be better. Here’s a hint: Suppose you have lotsof experience driving home from work. Then you move to a new building and a newparking lot (but you still enter the highway at the same place). Now you are starting
to learn predictions for the new building. Can you see why TD updates are likely to bemuch better, at least initially, in this case? Might the same sort of thing happen in theoriginal scenario?
⇤

124

6.2

Chapter 6: Temporal-Di↵erence Learning

Advantages of TD Prediction Methods

TD methods update their estimates based in part on other estimates. They learn aguess from a guess—they bootstrap. Is this a good thing to do? What advantages do
TD methods have over Monte Carlo and DP methods? Developing and answering suchquestions will take the rest of this book and more. In this section we briefly anticipate
some of the answers.
Obviously, TD methods have an advantage over DP methods in that they do not
require a model of the environment, of its reward and next-state probability distributions.
The next most obvious advantage of TD methods over Monte Carlo methods is thatthey are naturally implemented in an online, fully incremental fashion. With Monte
Carlo methods one must wait until the end of an episode, because only then is the returnknown, whereas with TD methods one need wait only one time step. Surprisingly often
this turns out to be a critical consideration. Some applications have very long episodes, sothat delaying all learning until the end of the episode is too slow. Other applications arecontinuing tasks and have no episodes at all. Finally, as we noted in the previous chapter,some Monte Carlo methods must ignore or discount episodes on which experimental
actions are taken, which can greatly slow learning. TD methods are much less susceptible
to these problems because they learn from each transition regardless of what subsequentactions are taken.
But are TD methods sound? Certainly it is convenient to learn one guess from thenext, without waiting for an actual outcome, but can we still guarantee convergenceto the correct answer? Happily, the answer is yes. For any fixed policy ⇡, TD(0) has
been proved to converge to v⇡ , in the mean for a constant step-size parameter if it issufficiently small, and with probability 1 if the step-size parameter decreases according to
the usual stochastic approximation conditions (2.7). Most convergence proofs apply only
to the table-based case of the algorithm presented above (6.2), but some also apply tothe case of general linear function approximation. These results are discussed in a moregeneral setting in Section 9.4.
If both TD and Monte Carlo methods converge asymptotically to the correct predictions,then a natural next question is “Which gets there first?” In other words, which method
learns faster? Which makes the more efficient use of limited data? At the current time
this is an open question in the sense that no one has been able to prove mathematicallythat one method converges faster than the other. In fact, it is not even clear what is themost appropriate formal way to phrase this question! In practice, however, TD methods
have usually been found to converge faster than constant-↵ MC methods on stochastic
tasks, as illustrated in Example 6.2.

6.2. Advantages of TD Prediction Methods

Example 6.2

125

Random WalkIn this example we empirically compare the prediction abilities of TD(0) andconstant-↵ MC when applied to the following Markov reward process:
0

A

0

0

B

C

0

0

D

1

E

start

A Markov reward process, or MRP, is a Markov decision process without actions.
We will often use MRPs when focusing on the prediction problem, in which there isno need to distinguish the dynamics due to the environment from those due to theagent. In this MRP, all episodes start in the center state, C, then proceed either leftor right by one state on each step, with equal probability. Episodes terminate either
on the extreme left or the extreme right. When an episode terminates on the right,a reward of +1 occurs; all other rewards are zero. For example, a typical episodemight consist of the following state-and-reward sequence: C, 0, B, 0, C, 0, D, 0, E, 1.
Because this task is undiscounted, the true value of each state is the probability ofterminating on the right if starting from that state. Thus, the true value of thecenter state is v⇡ (C) = 0.5. The true values of all the states, A through E, are
1 2 3 45
6 , 6 , 6 , 6 , and 6 .
Estimated
value

0.8

0.4

0.2

10

0.6

Estimated
value

Empirical RMS error,
averaged over states

0.25

100

0
1

RMS error,
averagedTrueover statesvalues
0.2

_ 

MC

0.15
_ 

_ 

0.1
0.05

0

_ 

_ 

TD

_ 

_ 

0

A

B

C

State

D

E

0

25

50

75

100

Walks / Episodes

The left graph above shows the values learned after various numbers of episodes ona single run of TD(0). The estimates after 100 episodes are about as close as they
ever come to the true values—with a constant step-size parameter (↵ = 0.1 in thisexample), the values fluctuate indefinitely in response to the outcomes of the mostrecent episodes. The right graph shows learning curves for the two methods forvarious values of ↵. The performance measure shown is the root mean square (RMS)
error between the value function learned and the true value function, averaged overthe five states, then averaged over 100 runs. In all cases the approximate valuefunction was initialized to the intermediate value V (s) = 0.5, for all s. The TD
method was consistently better than the MC method on this task.

126

Chapter 6: Temporal-Di↵erence Learning

Exercise 6.3 From the results shown in the left graph of the random walk example itappears that the first episode results in a change in only V (A). What does this tell you
about what happened on the first episode? Why was only the estimate for this one statechanged? By exactly how much was it changed?
⇤
Exercise 6.4 The specific results shown in the right graph of the random walk example
are dependent on the value of the step-size parameter, ↵. Do you think the conclusions
about which algorithm is better would be a↵ected if a wider range of ↵ values were used?
Is there a di↵erent, fixed value of ↵ at which either algorithm would have performed
significantly better than shown? Why or why not?
⇤⇤Exercise 6.5 In the right graph of the random walk example, the RMS error of the
TD method seems to go down and then up again, particularly at high ↵’s. What could
have caused this? Do you think this always occurs, or might it be a function of how theapproximate value function was initialized?
⇤
Exercise 6.6 In Example 6.2 we stated that the true values for the random walk example
are 16 , 26 , 36 , 46 , and 56 , for states A through E. Describe at least two di↵erent ways thatthese could have been computed. Which would you guess we actually used? Why? ⇤6.3

Optimality of TD(0)Suppose there is available only a finite amount of experience, say 10 episodes or 100
time steps. In this case, a common approach with incremental learning methods is topresent the experience repeatedly until the method converges upon an answer. Given an
approximate value function, V , the increments specified by (6.1) or (6.2) are computedfor every time step t at which a nonterminal state is visited, but the value function ischanged only once, by the sum of all the increments. Then all the available experience isprocessed again with the new value function to produce a new overall increment, and so
on, until the value function converges. We call this batch updating because updates are
made only after processing each complete batch of training data.
Under batch updating, TD(0) converges deterministically to a single answer independentof the step-size parameter, ↵, as long as ↵ is chosen to be sufficiently small. The constant↵ MC method also converges deterministically under the same conditions, but to adi↵erent answer. Understanding these two answers will help us understand the di↵erence
between the two methods. Under normal updating the methods do not move all the way
to their respective batch answers, but in some sense they take steps in these directions.
Before trying to understand the two answers in general, for all possible tasks, we firstlook at a few examples.
Example 6.3: Random walk under batch updating Batch-updating versions ofTD(0) and constant-↵ MC were applied as follows to the random walk prediction example(Example 6.2). After each new episode, all episodes seen so far were treated as a batch.
They were repeatedly presented to the algorithm, either TD(0) or constant-↵ MC, with↵ sufficiently small that the value function converged. The resulting value function wasthen compared with v⇡ , and the average root mean square error across the five states(and across 100 independent repetitions of the whole experiment) was plotted to obtain6.3. Optimality of TD(0)127

the learning curves shown in Figure 6.2. Note that the batch TD method was consistently
better than the batch Monte Carlo method.
Under batch training, constant-↵
MC converges to values, V (s), that.25
are sample averages of the actual reBATCH TRAINING.2
turns experienced after visiting eachstate s. These are optimal estimatesin the sense that they minimize the RMS error, .15
mean square error from the actual averaged
over states . 1
MC
returns in the training set. In thisTD
sense it is surprising that the batch.05
TD method was able to performbetter according to the root mean
.0
0
25
50
75
100
square error measure shown in theWalks / Episodes
figure to the right. How is it thatbatch TD was able to perform better
than this optimal method? The an- Figure 6.2: Performance of TD(0) and constant-↵swer is that the Monte Carlo method MC under batch training on the random walk task.
is optimal only in a limited way, andthat TD is optimal in a way that is more relevant to predicting returns.
Example 6.4: You are the Predictor Place yourself now in the role of the predictorof returns for an unknown Markov reward process. Suppose you observe the following
eight episodes:
A, 0, B, 0
B, 1
B, 1
B, 1

B, 1
B, 1
B, 1
B, 0

This means that the first episode started in state A, transitioned to B with a reward of0, and then terminated from B with a reward of 0. The other seven episodes were evenshorter, starting from B and terminating immediately. Given this batch of data, what
would you say are the optimal predictions, the best values for the estimates V (A) andV (B)? Everyone would probably agree that the optimal value for V (B) is 34 , because six
out of the eight times in state B the process terminated immediately with a return of 1,
and the other two times in B the process terminated immediately with a return of 0.
But what is the optimal value for the estimate V (A) given this data? Here there are
two reasonable answers. One is to observe that 100% of thetimes the process was in state A it traversed immediately to
r=1
B (with a reward of 0); and because we have already decided
75%
that B has value 34 , therefore A must have value 34 as well.
r= 0
A
BOne way of viewing this answer is that it is based on first
100%
r= 0
modeling the Markov process, in this case as shown to the25%
right, and then computing the correct estimates given themodel, which indeed in this case gives V (A) = 34 . This isalso the answer that batch TD(0) gives.

128

Chapter 6: Temporal-Di↵erence Learning

The other reasonable answer is simply to observe that we have seen A once and thereturn that followed it was 0; we therefore estimate V (A) as 0. This is the answer thatbatch Monte Carlo methods give. Notice that it is also the answer that gives minimum
squared error on the training data. In fact, it gives zero error on the data. But still we
expect the first answer to be better. If the process is Markov, we expect that the firstanswer will produce lower error on future data, even though the Monte Carlo answer isbetter on the existing data.
Example 6.4 illustrates a general di↵erence between the estimates found by batch
TD(0) and batch Monte Carlo methods. Batch Monte Carlo methods always find theestimates that minimize mean square error on the training set, whereas batch TD(0)always finds the estimates that would be exactly correct for the maximum-likelihood
model of the Markov process. In general, the maximum-likelihood estimate of a parameteris the parameter value whose probability of generating the data is greatest. In this case,the maximum-likelihood estimate is the model of the Markov process formed in theobvious way from the observed episodes: the estimated transition probability from i to jis the fraction of observed transitions from i that went to j, and the associated expected
reward is the average of the rewards observed on those transitions. Given this model,
we can compute the estimate of the value function that would be exactly correct if themodel were exactly correct. This is called the certainty-equivalence estimate because itis equivalent to assuming that the estimate of the underlying process was known withcertainty rather than being approximated. In general, batch TD(0) converges to thecertainty-equivalence estimate.
This helps explain why TD methods converge more quickly than Monte Carlo methods.
In batch form, TD(0) is faster than Monte Carlo methods because it computes thetrue certainty-equivalence estimate. This explains the advantage of TD(0) shown in thebatch results on the random walk task (Figure 6.2). The relationship to the certaintyequivalence estimate may also explain in part the speed advantage of nonbatch TD(0)(e.g., Example 6.2, page 125, right graph). Although the nonbatch methods do not achieve
either the certainty-equivalence or the minimum squared error estimates, they can beunderstood as moving roughly in these directions. Nonbatch TD(0) may be faster thanconstant-↵ MC because it is moving toward a better estimate, even though it is notgetting all the way there. At the current time nothing more definite can be said about
the relative efficiency of online TD and Monte Carlo methods.
Finally, it is worth noting that although the certainty-equivalence estimate is in somesense an optimal solution, it is almost never feasible to compute it directly. If n = |S| isthe number of states, then just forming the maximum-likelihood estimate of the process
may require on the order of n2 memory, and computing the corresponding value function
requires on the order of n3 computational steps if done conventionally. In these terms it
is indeed striking that TD methods can approximate the same solution using memory
no more than order n and repeated computations over the training set. On tasks withlarge state spaces, TD methods may be the only feasible way of approximating thecertainty-equivalence solution.
⇤

Exercise 6.7 Design an o↵-policy version of the TD(0) update that can be used with
arbitrary target policy ⇡ and covering behavior policy b, using at each step t the importancesampling ratio ⇢t:t (5.3).
⇤

6.4. Sarsa: On-policy TD Control

6.4

129

Sarsa: On-policy TD Control

We turn now to the use of TD prediction methods for the control problem. As usual, we
follow the pattern of generalized policy iteration (GPI), only this time using TD methods
for the evaluation or prediction part. As with Monte Carlo methods, we face the need to
trade o↵ exploration and exploitation, and again approaches fall into two main classes:
on-policy and o↵-policy. In this section we present an on-policy TD control method.
The first step is to learn an action-value function rather than a state-value function.
In particular, for an on-policy method we must estimate q⇡ (s, a) for the current behavior
policy ⇡ and for all states s and actions a. This can be done using essentially the same TD
method described above for learning v⇡ . Recall that an episode consists of an alternating
sequence of states and state–action pairs:

...

St

Rt+1
At

St+1

Rt+2
Rt+3
St+2
St+3
At+1
At+2
At+3

...

In the previous section we considered transitions from state to state and learned thevalues of states. Now we consider transitions from state–action pair to state–action pair,and learn the values of state–action pairs. Formally these cases are identical: they areboth Markov chains with a reward process. The theorems assuring the convergence ofstate values under TD(0) also apply to the corresponding algorithm for action values:hi
Q(St , At )Q(St , At ) + ↵ Rt+1 + Q(St+1 , At+1 ) Q(St , At ) .
(6.7)

This update is done after every transition from a nonterminal state St . If
St+1 is terminal, then Q(St+1 , At+1 ) is defined as zero. This rule uses every
element of the quintuple of events, (St , At , Rt+1 , St+1 , At+1 ), that make up atransition from one state–action pair to the next. This quintuple gives rise to
the name Sarsa for the algorithm. The backup diagram for Sarsa is as shown
Sarsa
to the right.
It is straightforward to design an on-policy control algorithm based on the Sarsa
prediction method. As in all on-policy methods, we continually estimate q⇡ for the
behavior policy ⇡, and at the same time change ⇡ toward greediness with respect to q⇡ .
The general form of the Sarsa control algorithm is given in the box on the next page.
The convergence properties of the Sarsa algorithm depend on the nature of the policy’s
dependence on Q. For example, one could use "-greedy or "-soft policies. Sarsa converges
with probability 1 to an optimal policy and action-value function, under the usual
conditions on the step sizes (2.7), as long as all state–action pairs are visited an infinite
number of times and the policy converges in the limit to the greedy policy (which can bearranged, for example, with "-greedy policies by setting " = 1/t).
Exercise 6.8 Show that an action-value version of (6.6) holds for the action-value formof the TD error t = Rt+1 + Q(St+1 , At+1 ) Q(St , At ), again assuming that the values
don’t change from step to step.
⇤

130

Chapter 6: Temporal-Di↵erence Learning

Sarsa (on-policy TD control) for estimating Q ⇡ q⇤
Algorithm parameters: step size ↵ 2 (0, 1], small " > 0
Initialize Q(s, a), for all s 2 S+ , a 2 A(s), arbitrarily except that Q(terminal , ·) = 0
Loop for each episode:
Initialize S
Choose A from S using policy derived from Q (e.g., "-greedy)
Loop for each step of episode:Take action A, observe R, S 0
Choose A0 from S 0 using⇥ policy derived from Q (e.g.,⇤ "-greedy)
Q(S, A)
Q(S, A) + ↵ R + Q(S 0 , A0 ) Q(S, A)
S
S0; A
A0 ;
until S is terminal

EpisodesExample 6.5: Windy Gridworld Shown inset below is a standard gridworld, withstart and goal states, but with one di↵erence: there is a crosswind running upward
through the middle of the grid. The actions are the standard four—up, down, right,
and left—but in the middle region the resultant next states are shifted upward by a“wind,” the strength of which varies from column to column. The strength of the windis given below each column, in number of cells shifted upward. For ex170
ample, if you are one cell to the150
right of the goal, then the actionG
Sleft takes you to the cell just above
Actions
the goal. This is an undiscounted
100
episodic task, with constant rewards
0 0 0 1 1 1 2 2 1 0
of 1 until the goal state is reached.
50
The graph to the right shows theresults of applying "-greedy Sarsa tothis task, with " = 0.1, ↵ = 0.5,
0and the initial values Q(s, a) = 001000 2000 3000 4000 5000 6000 7000 8000
for all s, a. The increasing slope ofTime steps
the graph shows that the goal wasreached more quickly over time. By
8000 time steps, the greedy policy was long since optimal (a trajectory from it is shown
inset); continued "-greedy exploration kept the average episode length at about 17 steps,
two more than the minimum of 15. Note that Monte Carlo methods cannot easily be
used here because termination is not guaranteed for all policies. If a policy was everfound that caused the agent to stay in the same state, then the next episode would
never end. Online learning methods such as Sarsa do not have this problem because theyquickly learn during the episode that such policies are poor, and switch to somethingelse.

6.5. Q-learning: O↵-policy TD Control

131

Exercise 6.9: Windy Gridworld with King’s Moves (programming) Re-solve the windy
gridworld assuming eight possible actions, including the diagonal moves, rather than four.
How much better can you do with the extra actions? Can you do even better by including
a ninth action that causes no movement at all other than that caused by the wind? ⇤Exercise 6.10: Stochastic Wind (programming) Re-solve the windy gridworld task with
King’s moves, assuming that the e↵ect of the wind, if there is any, is stochastic, sometimes
varying by 1 from the mean values given for each column. That is, a third of the time
you move exactly according to these values, as in the previous exercise, but also a thirdof the time you move one cell above that, and another third of the time you move one
cell below that. For example, if you are one cell to the right of the goal and you moveleft, then one-third of the time you move one cell above the goal, one-third of the time
you move two cells above the goal, and one-third of the time you move to the goal. ⇤6.5

Q-learning: O↵-policy TD Control

One of the early breakthroughs in reinforcement learning was the development of ano↵-policy TD control algorithm known as Q-learning (Watkins, 1989), defined byhiQ(St , At )Q(St , At ) + ↵ Rt+1 + max Q(St+1 , a) Q(St , At ) .
(6.8)
a

In this case, the learned action-value function, Q, directly approximates q⇤ , the optimal
action-value function, independent of the policy being followed. This dramaticallysimplifies the analysis of the algorithm and enabled early convergence proofs. The policy
still has an e↵ect in that it determines which state–action pairs are visited and updated.
However, all that is required for correct convergence is that all pairs continue to be
updated. As we observed in Chapter 5, this is a minimal requirement in the sense thatany method guaranteed to find optimal behavior in the general case must require it.
Under this assumption and a variant of the usual stochastic approximation conditions on
the sequence of step-size parameters, Q has been shown to converge with probability 1 toq⇤ . The Q-learning algorithm is shown below in procedural form.
Q-learning (o↵-policy TD control) for estimating ⇡ ⇡ ⇡⇤Algorithm parameters: step size ↵ 2 (0, 1], small " > 0
Initialize Q(s, a), for all s 2 S+ , a 2 A(s), arbitrarily except that Q(terminal , ·) = 0
Loop for each episode:
Initialize S
Loop for each step of episode:
Choose A from S using policy derived from Q (e.g., "-greedy)
Take action A, observe ⇥R, S 0⇤
Q(S, A)
Q(S, A) + ↵ R + maxa Q(S 0 , a) Q(S, A)
S
S0
until S is terminal

132

Chapter 6: Temporal-Di↵erence Learning

What is the backup diagram for Q-learning? The rule (6.8) updates a state–action
pair, so the top node, the root of the update, must be a small, filled action node. The
update is also from action nodes, maximizing over all those actions possible in the next
state. Thus the bottom nodes of the backup diagram should be all these action nodes.
Finally, remember that we indicate taking the maximum of these “next action” nodes
with an arc across them (Figure 3.4-right). Can you guess now what the diagram is? If
so, please do make a guess before turning to the answer in Figure 6.4 on page 134.
Example 6.6: Cli↵ Walking This gridworld example compares Sarsa and Q-learning,highlighting the di↵erence between on-policy (Sarsa) and o↵-policy (Q-learning) methods.
Consider the gridworld shown to theright. This is a standard undisR = -1
counted, episodic task, with start
SaferRpath
and goal states, and the usual actions causing movement up, down,
Optimal pathright, and left. Reward is 1 on all
R = -1T h e C l i f fSGtransitions except those into the re- safe Rpathr = sagion marked “The Cli↵.” Stepping
R = -100
into this region incurs a reward ofop
optimal path
100 and sends the agent instantly
T
h
e
C
l
i
f
f
S
G
back to the start.
The graph to the right shows theSarsa
R = -100
performance of the Sarsa and QRr=   
learning methods with "-greedy action selection, " = 0.1. After an
Sum of  
initial transient, Q-learning learns RewardSarsa
rewards
Q-learning
values for the optimal policy, that per -25
duringwhich travels right along the edge epsiode
episodeof the cli↵. Unfortunately, this re sults in its occasionally falling o↵ Sum of -50
rewards
Q-learning
the cli↵ because of the "-greedy acduringtion selection. Sarsa, on the other episode



~

-75 
hand, takes the action selection into
EpisodesEpisodesaccount and learns the longer but
safer path through the upper part-100of the grid. Although Q-learning ac0
100
200
300
400
500
tually learns the values of the optiEpisodes
mal policy, its online performanceis worse than that of Sarsa, whichlearns the roundabout policy. Of course, if " were gradually reduced, then both methods
would asymptotically converge to the optimal policy.
Exercise 6.11 Why is Q-learning considered an o↵-policy control method?

⇤

Exercise 6.12 Suppose action selection is greedy. Is Q-learning then exactly the samealgorithm as Sarsa? Will they make exactly the same action selections and weight
updates?
⇤6.6. Expected Sarsa

6.6

133

Expected Sarsa

Consider the learning algorithm that is just like Q-learning except that instead of the
maximum over next state–action pairs it uses the expected value, taking into account
how likely each action is under the current policy. That is, consider the algorithm withthe update rule
h
i
Q(S
,
A
)
Q(S
,
A
)
+
↵
R
+
E
[Q(S
,
A
)
|
S
]
Q(S
,
A
)
t
t
t
t
t+1
⇡
t+1
t+1
t+1
t
t
ts on two versions of the windy on the edge of the cliff immediately, resulting in a slightlyih performance.
Xwith a deterministic environment better on-line= Q(St , At ) + ↵ Rt+1 +⇡(a|St+1 )Q(St+1 , a) Q(St , At ) ,(6.9)
c environment. We do so in order
For n = 100, 000, the average return is equal for allae of environment stochasticity on ↵ values in case of Expected Sarsa and Q-learning. Thisbut thatotherwise
follows theof Q-learning.
Given thenext
state,
nce between ExpectedSarsa
and indicates
thatschema
the algorithms
have converged
longbeforetheSt+1 , thisalgorithm
in thedirection
Sarsa
moves
expectation,
rst part of Hypothesis
2. We moves
then deterministicallyend of the run for
all same↵ values,
sinceaswedo notseeinany
andstochasticity
accordingly it iscalledItsphase.
backupshown on the right in
ent amounts of policy
effectof Expected
the initialSarsa.
learning
Fordiagram
Sarsa theis performance
Figure6.4.
of Hypothesis 2. For
completeness,comes close to the performance of Expected Sarsa only for
ExpectedSarsa
iscomplexcomputationallythan Sarsa but,it eliminates
ance of Q-learning on this problem. ↵more= 0.1,
while forlarge ↵, the performancefor nin=return,100, 000
thevariancedue
tothe
randomselection
of
A
.
Given
the
same
amountof
t+1
ts in other domains verifying the even drops below the performance for n = 100. The reasonexperience
we might
expect it istothat
perform
slightly
betterSarsa,
andit generally does.
arsa in a broader setting.
All results
for large
valuesof ↵thanthe Qvaluesof indeed
Sarsa diverge.
Figure6.3
showssummary
resultsonthecli↵-walking
task
with
Expected
Sarsa compared
raged over numerous independent Although the policy is still improved over the initial random
to Sarsa and Q-learning. Expected Sarsa retains the significant advantage of Sarsa overard error becomes negligible.
policy during the early stages of learning, divergence causes
Q-learning on this problem. In addition, Expected Sarsa shows a significant improvement
the policy to get worse in the long run.

0
ypothesis 1 using the cliff walking
isodic navigation task in which the−20from start to goal in a deterministic−40
ge of the grid world is a cliff (see-40take any of four movement actions:rewards −60
each of which moves theSum
agentofone
perepisodeng direction. Each step results in a−80
-80
n the agent steps into the cliff area,d of -100 and an immediate return−100
isode ends upon reaching the goal
average return0

Q-learning

Sarsa
Q-learningn = 100, Sarsa

Interim Performancen = 100, Q−learning

-120
−120

n = 100, Expected Sarsa
n = 1E5, Sarsa
n = 1E5, Q−learningn = 1E5, Expected Sarsa

−140

−160
0.1
0.1

Expected Sarsa

Asymptotic Performance

0.2
0.2

0.3
0.3

0.4
0.4

0.5
0.5

↵

0.6
0.6

alpha

0.7
0.7

0.8
0.8

0.9
0.9

11

G
Figure6.3: Interim and asymptotic performance of TD control methods on the cli↵-walking
Fig. 2. Average return on the cliff walking task over the first n episodes
task as a function offor↵.n =All
used"-greedy
policywith
0.1.The
Asymptotic
100algorithms
and n = 100,000 anusing
an -greedy
policywith" =
= 0.1.
. The agent has to movefrom the startis[S]an average
performance
100,000
episodeswhereas interim performance is an average
big dots overindicate
the maximal
values.
stepping into the cliff (greyover area).
the first 100 episodes. These data are averages of over 50,000 and 10 runs for the interim
and asymptotic cases respectively. The solid circles mark the best interim performance of eachmethod.
AdaptedSeijenet World
al. (2009).
B. vanWindy
Gridrmance over the firstn episodes
as fromg rate ↵ using an -greedy policy
hows the result for n = 100 anded the results over 50,000 runs andWe turn to the windy grid world task to further test Hypothesis 2. The windy grid world task is another navigationtask, where the agent has to find its way from start to goal.

134

Chapter 6: Temporal-Di↵erence Learning

Q-learning

Expected Sarsa

Figure 6.4: The backup diagrams for Q-learning and Expected Sarsa.

over Sarsa over a wide range of values for the step-size parameter ↵. In cli↵ walking
the state transitions are all deterministic and all randomness comes from the policy. Insuch cases, Expected Sarsa can safely set ↵ = 1 without su↵ering any degradation of
asymptotic performance, whereas Sarsa can only perform well in the long run at a smallvalue of ↵, at which short-term performance is poor. In this and other examples there isa consistent empirical advantage of Expected Sarsa over Sarsa.
In these cli↵ walking results Expected Sarsa was used on-policy, but in general it
might use a policy di↵erent from the target policy ⇡ to generate behavior, in which caseit becomes an o↵-policy algorithm. For example, suppose ⇡ is the greedy policy while
behavior is more exploratory; then Expected Sarsa is exactly Q-learning. In this sense
Expected Sarsa subsumes and generalizes Q-learning while reliably improving over Sarsa.
Except for the small additional computational cost, Expected Sarsa may completely
dominate both of the other more-well-known TD control algorithms.

6.7

Maximization Bias and Double Learning

All the control algorithms that we have discussed so far involve maximization in theconstruction of their target policies. For example, in Q-learning the target policy is
the greedy policy given the current action values, which is defined with a max, and in
Sarsa the policy is often "-greedy, which also involves a maximization operation. In thesealgorithms, a maximum over estimated values is used implicitly as an estimate of the
maximum value, which can lead to a significant positive bias. To see why, consider asingle state s where there are many actions a whose true values, q(s, a), are all zero but
whose estimated values, Q(s, a), are uncertain and thus distributed some above and somebelow zero. The maximum of the true values is zero, but the maximum of the estimatesis positive, a positive bias. We call this maximization bias.
Example 6.7: Maximization Bias Example The small MDP shown inset in
Figure 6.5 provides a simple example of how maximization bias can harm the performanceof TD control algorithms. The MDP has two non-terminal states A and B. Episodes
always start in A with a choice between two actions, left and right. The right action
transitions immediately to the terminal state with a reward and return of zero. The
left action transitions to B, also with a reward of zero, from which there are many
possible actions all of which cause immediate termination with a reward drawn from anormal distribution with mean 0.1 and variance 1.0. Thus, the expected return for
any trajectory starting with left is 0.1, and thus taking left in state A is always a6.7. Maximization Bias and Double Learning100%

N( 0.1, 1)

...

75%

% leftactionsfrom A

135

50%

B

0
left

A

0
right

Q-learning
DoubleQ-learning

25%

optimal

5%
0
1

100

200

300

EpisodesFigure 6.5: Comparison of Q-learning and Double Q-learning on a simple episodic MDP (shown
inset). Q-learning initially learns to take the left action much more often than the right action,
and always takes it significantly more often than the 5% minimum probability enforced by"-greedy action selection with " = 0.1. In contrast, Double Q-learning is essentially una↵ected by
maximization bias. These data are averaged over 10,000 runs. The initial action-value estimates
were zero. Any ties in "-greedy action selection were broken randomly.

mistake. Nevertheless, our control methods may favor left because of maximization bias
making B appear to have a positive value. Figure 6.5 shows that Q-learning with "-greedy
action selection initially learns to strongly favor the left action on this example. Even at
asymptote, Q-learning takes the left action about 5% more often than is optimal at ourparameter settings (" = 0.1, ↵ = 0.1, and = 1).
Are there algorithms that avoid maximization bias? To start, consider a bandit case inwhich we have noisy estimates of the value of each of many actions, obtained as sample
averages of the rewards received on all the plays with each action. As we discussed above,
there will be a positive maximization bias if we use the maximum of the estimates as
an estimate of the maximum of the true values. One way to view the problem is that
it is due to using the same samples (plays) both to determine the maximizing action
and to estimate its value. Suppose we divided the plays in two sets and used them tolearn two independent estimates, call them Q1 (a) and Q2 (a), each an estimate of thetrue value q(a), for all a 2 A. We could then use one estimate, say Q1 , to determine
the maximizing action A⇤ = argmaxa Q1 (a), and the other, Q2 , to provide the estimate
of its value, Q2 (A⇤ ) = Q2 (argmaxa Q1 (a)). This estimate will then be unbiased in thesense that E[Q2 (A⇤ )] = q(A⇤ ). We can also repeat the process with the role of the two
estimates reversed to yield a second unbiased estimate Q1 (argmaxa Q2 (a)). This is theidea of double learning. Note that although we learn two estimates, only one estimate is
updated on each play; double learning doubles the memory requirements, but does notincrease the amount of computation per step.
The idea of double learning extends naturally to algorithms for full MDPs. For example,
the double learning algorithm analogous to Q-learning, called Double Q-learning, divides
the time steps in two, perhaps by flipping a coin on each step. If the coin comes up heads,
the update is

136

Chapter 6: Temporal-Di↵erence Learning

Q1 (St , At )h
Q1 (St , At )+↵ Rt+1 + Q2 St+1 , argmax Q1 (St+1 , a)ai
Q1 (St , At ) . (6.10)

If the coin comes up tails, then the same update is done with Q1 and Q2 switched,so that Q2 is updated. The two approximate value functions are treated completely
symmetrically. The behavior policy can use both action-value estimates. For example, an"-greedy policy for Double Q-learning could be based on the average (or sum) of the two
action-value estimates. A complete algorithm for Double Q-learning is given in the box
below. This is the algorithm used to produce the results in Figure 6.5. In that example,
double learning seems to eliminate the harm caused by maximization bias. Of course
there are also double versions of Sarsa and Expected Sarsa.
Double Q-learning, for estimating Q1 ⇡ Q2 ⇡ q⇤
Algorithm parameters: step size ↵ 2 (0, 1], small " > 0
Initialize Q1 (s, a) and Q2 (s, a), for all s 2 S+ , a 2 A(s), such that Q(terminal , ·) = 0
Loop for each episode:
Initialize S
Loop for each step of episode:
Choose A from S using the policy "-greedy in Q1 + Q2
Take action A, observe R, S 0
With 0.5 probabilility:⇣
Q1 (S, A)else:

Q1 (S, A) + ↵ R + Q2 S 0 , argmaxa Q1 (S 0 , a)⇣
Q2 (S, A)
Q2 (S, A) + ↵ R + Q1 S 0 , argmaxa Q2 (S 0 , a)S
S0until S is terminal

Q1 (S, A)
Q2 (S, A)⌘⌘⇤

Exercise 6.13 What are the update equations for Double Expected Sarsa with an"-greedy target policy?
⇤

6.8

Games, Afterstates, and Other Special Cases

In this book we try to present a uniform approach to a wide class of tasks, but ofcourse there are always exceptional tasks that are better treated in a specialized way. For
example, our general approach involves learning an action-value function, but in Chapter 1
we presented a TD method for learning to play tic-tac-toe that learned something much
more like a state-value function. If we look closely at that example, it becomes apparent
that the function learned there is neither an action-value function nor a state-valuefunction in the usual sense. A conventional state-value function evaluates states in which
the agent has the option of selecting an action, but the state-value function used in6.8. Games, Afterstates, and Other Special Cases

137

tic-tac-toe evaluates board positions after the agent has made its move. Let us call theseafterstates, and value functions over these, afterstate value functions. Afterstates areuseful when we have knowledge of an initial part of the environment’s dynamics but notnecessarily of the full dynamics. For example, in games we typically know the immediatee↵ects of our moves. We know for each possible chess move what the resulting position
will be, but not how our opponent will reply. Afterstate value functions are a natural
way to take advantage of this kind of knowledge and thereby produce a more efficientlearning method.
The reason it is more efficient to design algorithms in terms of afterstates is apparent from the tic-tac-toe example. A conventional action-value function would map
from positions and moves to an estimate of the value. But many position—move
pairs produce the same resulting position, as in the example below: In such cases the
position–move pairs are di↵erent but produce the same “afterposition,” and thus must have X
X
the same value. A conventional
X
O
O X
action-value function would haveto separately assess both pairs,
whereas an afterstate value function would immediately assess
both equally. Any learning about
the position–move pair on the left
would immediately transfer to theXpair on the right.
O X
Afterstates arise in many tasks,
not just games. For example, inqueuing tasks there are actions
such as assigning customers to servers, rejecting customers, or discarding information. In
such cases the actions are in fact defined in terms of their immediate e↵ects, which arecompletely known.
It is impossible to describe all the possible kinds of specialized problems and corresponding specialized learning algorithms. However, the principles developed in this bookshould apply widely. For example, afterstate methods are still aptly described in termsof generalized policy iteration, with a policy and (afterstate) value function interacting inessentially the same way. In many cases one will still face the choice between on-policy
and o↵-policy methods for managing the need for persistent exploration.

++

Exercise 6.14 Describe how the task of Jack’s Car Rental (Example 4.2) could bereformulated in terms of afterstates. Why, in terms of this specific task, would such areformulation be likely to speed convergence?
⇤

138

6.9

Chapter 6: Temporal-Di↵erence Learning

Summary

In this chapter we introduced a new kind of learning method, temporal-di↵erence (TD)learning, and showed how it can be applied to the reinforcement learning problem. As
usual, we divided the overall problem into a prediction problem and a control problem.
TD methods are alternatives to Monte Carlo methods for solving the prediction problem.
In both cases, the extension to the control problem is via the idea of generalized policy
iteration (GPI) that we abstracted from dynamic programming. This is the idea thatapproximate policy and value functions should interact in such a way that they bothmove toward their optimal values.
One of the two processes making up GPI drives the value function to accurately predict
returns for the current policy; this is the prediction problem. The other process drives
the policy to improve locally (e.g., to be "-greedy) with respect to the current valuefunction. When the first process is based on experience, a complication arises concerning
maintaining sufficient exploration. We can classify TD control methods according towhether they deal with this complication by using an on-policy or o↵-policy approach.
Sarsa is an on-policy method, and Q-learning is an o↵-policy method. Expected Sarsa
is also an o↵-policy method as we present it here. There is a third way in which TD
methods can be extended to control which we did not include in this chapter, called
actor–critic methods. These methods are covered in full in Chapter 13.
The methods presented in this chapter are today the most widely used reinforcement
learning methods. This is probably due to their great simplicity: they can be applied
online, with a minimal amount of computation, to experience generated from interaction
with an environment; they can be expressed nearly completely by single equations thatcan be implemented with small computer programs. In the next few chapters we extend
these algorithms, making them slightly more complicated and significantly more powerful.
All the new algorithms will retain the essence of those introduced here: they will be able
to process experience online, with relatively little computation, and they will be driven
by TD errors. The special cases of TD methods introduced in the present chapter shouldrightly be called one-step, tabular, model-free TD methods. In the next two chapters we
extend them to n-step forms (a link to Monte Carlo methods) and forms that include
a model of the environment (a link to planning and dynamic programming). Then, inthe second part of the book we extend them to various forms of function approximation
rather than tables (a link to deep learning and artificial neural networks).
Finally, in this chapter we have discussed TD methods entirely within the context of
reinforcement learning problems, but TD methods are actually more general than this.
They are general methods for learning to make long-term predictions about dynamicalsystems. For example, TD methods may be relevant to predicting financial data, lifespans, election outcomes, weather patterns, animal behavior, demands on power stations,or customer purchases. It was only when TD methods were analyzed as pure prediction
methods, independent of their use in reinforcement learning, that their theoreticalproperties first came to be well understood. Even so, these other potential applicationsof TD learning methods have not yet been extensively explored.

6.9. Summary

139

Bibliographical and Historical Remarks
As we outlined in Chapter 1, the idea of TD learning has its early roots in animal learning
psychology and artificial intelligence, most notably the work of Samuel (1959) and Klopf(1972). Samuel’s work is described as a case study in Section 16.2. Also related to TD
learning are Holland’s (1975, 1976) early ideas about consistency among value predictions.
These influenced one of the authors (Barto), who was a graduate student from 1970 to1975 at the University of Michigan, where Holland was teaching. Holland’s ideas led toa number of TD-related systems, including the work of Booker (1982) and the bucket
brigade of Holland (1986), which is related to Sarsa as discussed below.
6.1–2 Most of the specific material from these sections is from Sutton (1988), including the TD(0) algorithm, the random walk example, and the term “temporaldi↵erence learning.” The characterization of the relationship to dynamic programming and Monte Carlo methods was influenced by Watkins (1989), Werbos(1987), and others. The use of backup diagrams was new to the first edition ofthis book.
Tabular TD(0) was proved to converge in the mean by Sutton (1988) and withprobability 1 by Dayan (1992), based on the work of Watkins and Dayan (1992).
These results were extended and strengthened by Jaakkola, Jordan, and Singh(1994) and Tsitsiklis (1994) by using extensions of the powerful existing theoryof stochastic approximation. Other extensions and generalizations are covered inlater chapters.
6.3

The optimality of the TD algorithm under batch training was established by
Sutton (1988). Illuminating this result is Barnard’s (1993) derivation of the TD
algorithm as a combination of one step of an incremental method for learning amodel of the Markov chain and one step of a method for computing predictions
from the model. The term certainty equivalence is from the adaptive control
literature (e.g., Goodwin and Sin, 1984).

6.4

The Sarsa algorithm was introduced by Rummery and Niranjan (1994). They
explored it in conjunction with artificial neural networks and called it “Modified
Connectionist Q-learning”. The name “Sarsa” was introduced by Sutton (1996).
The convergence of one-step tabular Sarsa (the form treated in this chapter) has
been proved by Singh, Jaakkola, Littman, and Szepesvári (2000). The “windy
gridworld” example was suggested by Tom Kalt.
Holland’s (1986) bucket brigade idea evolved into an algorithm closely related toSarsa. The original idea of the bucket brigade involved chains of rules triggering
each other; it focused on passing credit back from the current rule to the rulesthat triggered it. Over time, the bucket brigade came to be more like TD learningin passing credit back to any temporally preceding rule, not just to the ones
that triggered the current rule. The modern form of the bucket brigade, whensimplified in various natural ways, is nearly identical to one-step Sarsa, as detailed
by Wilson (1994).

140

Chapter 6: Temporal-Di↵erence Learning

6.5

Q-learning was introduced by Watkins (1989), whose outline of a convergence
proof was made rigorous by Watkins and Dayan (1992). More general convergence
results were proved by Jaakkola, Jordan, and Singh (1994) and Tsitsiklis (1994).

6.6

The Expected Sarsa algorithm was introduced by George John (1994), whocalled it “Q-learning” and stressed its advantages over Q-learning as an o↵-policy
algorithm. John’s work was not known to us when we presented Expected
Sarsa in the first edition of this book as an exercise, or to van Seijen, vanHasselt, Whiteson, and Weiring (2009) when they established Expected Sarsa’s
convergence properties and conditions under which it will outperform regularSarsa and Q-learning. Our Figure 6.3 is adapted from their results. Van Seijen
et al. defined “Expected Sarsa” to be an on-policy method exclusively (as we
did in the first edition), whereas now we use this name for the general algorithm
in which the target and behavior policies may di↵er. The general o↵-policy
view of Expected Sarsa was noted by van Hasselt (2011), who called it “General
Q-learning.”

6.7

Maximization bias and double learning were introduced and extensively investigated by van Hasselt (2010, 2011). The example MDP in Figure 6.5 was adaptedfrom that in his Figure 4.1 (van Hasselt, 2011).

6.8

The notion of an afterstate is the same as that of a “post-decision state” (VanRoy, Bertsekas, Lee, and Tsitsiklis, 1997; Powell, 2011).

Chapter 7

n-step Bootstrapping
In this chapter we unify the Monte Carlo (MC) methods and the one-step temporaldi↵erence (TD) methods presented in the previous two chapters. Neither MC methods nor
one-step TD methods are always the best. In this chapter we present n-step TD methods
that generalize both methods so that one can shift from one to the other smoothly as
needed to meet the demands of a particular task. n-step methods span a spectrum with
MC methods at one end and one-step TD methods at the other. The best methods areoften intermediate between the two extremes.
Another way of looking at the benefits of n-step methods is that they free you fromthe tyranny of the time step. With one-step TD methods the same time step determines
how often the action can be changed and the time interval over which bootstrapping
is done. In many applications one wants to be able to update the action very fast to
take into account anything that has changed, but bootstrapping works best if it is over alength of time in which a significant and recognizable state change has occurred. With
one-step TD methods, these time intervals are the same, and so a compromise must be
made. n-step methods enable bootstrapping to occur over multiple steps, freeing us fromthe tyranny of the single time step.
The idea of n-step methods is usually used as an introduction to the algorithmic
idea of eligibility traces (Chapter 12), which enable bootstrapping over multiple timeintervals simultaneously. Here we instead consider the n-step bootstrapping idea on its
own, postponing the treatment of eligibility-trace mechanisms until later. This allows us
to separate the issues better, dealing with as many of them as possible in the simpler
n-step setting.
As usual, we first consider the prediction problem and then the control problem. Thatis, we first consider how n-step methods can help in predicting returns as a function ofstate for a fixed policy (i.e., in estimating v⇡ ). Then we extend the ideas to action values
and control methods.

142

7.1

Chapter 7: n-step Bootstrappingn-step TD Prediction

What is the space of methods lying between Monte Carlo and TD methods? Consider
estimating v⇡ from sample episodes generated using ⇡. Monte Carlo methods perform
an update for each state based on the entire sequence of observed rewards from thatstate until the end of the episode. The update of one-step TD methods, on the other
hand, is based on just the one next reward, bootstrapping from the value of the state
one step later as a proxy for the remaining rewards. One kind of intermediate method,
then, would perform an update based on an intermediate number of rewards: more thanone, but less than all of them until termination. For example, a two-step update would
be based on the first two rewards and the estimated value of the state two steps later.
Similarly, we could have three-step updates, four-step updates, and so on. Figure 7.1
shows the backup diagrams of the spectrum of n-step updates for v⇡ , with the one-step
TD update on the left and the up-until-termination Monte Carlo update on the right.
1-step TD
and TD(0)2-step TD

3-step TD

n-step TD···∞-step TD
and Monte Carlo

···
···
···

Figure 7.1: The backup diagrams of n-step methods. These methods form a spectrum ranging
from one-step TD methods to Monte Carlo methods.

The methods that use n-step updates are still TD methods because they still change
an earlier estimate based on how it di↵ers from a later estimate. Now the later estimate
is not one step later, but n steps later. Methods in which the temporal di↵erence extends
over n steps are called n-step TD methods. The TD methods introduced in the previous
chapter all used one-step updates, which is why we called them one-step TD methods.
More formally, consider the update of the estimated value of state St as a result of thestate–reward sequence, St , Rt+1 , St+1 , Rt+2 , . . . , RT , ST (omitting the actions). We knowthat in Monte Carlo updates the estimate of v⇡ (St ) is updated in the direction of the7.1. n-step TD Prediction

complete return:.
Gt = Rt+1 + Rt+2 +143

2

Rt+3 + · · · +

T

t 1

RT ,

where T is the last time step of the episode. Let us call this quantity the target of the
update. Whereas in Monte Carlo updates the target is the return, in one-step updates
the target is the first reward plus the discounted estimated value of the next state, which
we call the one-step return:.
Gt:t+1 = Rt+1 + Vt (St+1 ),where Vt : S ! R here is the estimate at time t of v⇡ . The subscripts on Gt:t+1 indicatethat it is a truncated return for time t using rewards up until time t+1, with the discounted
estimate Vt (St+1 ) taking the place of the other terms Rt+2 + 2 Rt+3 + · · · + T t 1 RTof the full return, as discussed in the previous chapter. Our point now is that this idea
makes just as much sense after two steps as it does after one. The target for a two-step
update is the two-step return:.
Gt:t+2 = Rt+1 + Rt+2 + 2 Vt+1 (St+2 ),where now 2 Vt+1 (St+2 ) corrects for the absence of the terms 2 Rt+3 + 3 Rt+4 + · · · +T t 1
RT . Similarly, the target for an arbitrary n-step update is the n-step return:.
Gt:t+n = Rt+1 + Rt+2 + · · · + n 1 Rt+n + n Vt+n 1 (St+n ),(7.1)
for all n, t such that n 1 and 0  t < T n. All n-step returns can be consideredapproximations to the full return, truncated after n steps and then corrected for theremaining missing terms by Vt+n 1 (St+n ). If t + n T (if the n-step return extends
to or beyond termination), then all the missing terms are taken as zero, and the n-step.
return defined to be equal to the ordinary full return (Gt:t+n = Gt if t + n T ).
Note that n-step returns for n > 1 involve future rewards and states that are notavailable at the time of transition from t to t + 1. No real algorithm can use the n-stepreturn until after it has seen Rt+n and computed Vt+n 1 . The first time these areavailable is t + n. The natural state-value learning algorithm for using n-step returns is
thus
⇥
⇤
.
Vt+n (St ) = Vt+n 1 (St ) + ↵ Gt:t+n Vt+n 1 (St ) ,0  t < T,
(7.2)

while the values of all other states remain unchanged: Vt+n (s) = Vt+n 1 (s), for all s 6= St .
We call this algorithm n-step TD. Note that no changes at all are made during the firstn 1 steps of each episode. To make up for that, an equal number of additional updates
are made at the end of the episode, after termination and before starting the next episode.
Complete pseudocode is given in the box on the next page.
Exercise 7.1 In Chapter 6 we noted that the Monte Carlo error can be written as thesum of TD errors (6.6) if the value estimates don’t change from step to step. Show thatthe n-step error used in (7.2) can also be written as a sum of TD errors (again if thevalue estimates don’t change) generalizing the earlier result.
⇤
Exercise 7.2 (programming) With an n-step method, the value estimates do change fromstep to step, so an algorithm that used the sum of TD errors (see previous exercise) in

144

Chapter 7: n-step Bootstrappingn-step TD for estimating V ⇡ v⇡
Input: a policy ⇡Algorithm parameters: step size ↵ 2 (0, 1], a positive integer n
Initialize V (s) arbitrarily, for all s 2 S
All store and access operations (for St and Rt ) can take their index mod n + 1
Loop for each episode:
Initialize and store S0 6= terminal
T
1
Loop for t = 0, 1, 2, . . . :| If t < T , then:|Take an action according to ⇡(·|St )|Observe and store the next reward as Rt+1 and the next state as St+1|
If St+1 is terminal, then Tt+1| ⌧t n + 1 (⌧ is the time whose state’s estimate is being updated)| If ⌧ 0:
Pmin(⌧ +n,T ) i ⌧ 1|
GRi
i=⌧ +1|
If ⌧ + n < T , then: G
G + n V (S⌧ +n )(G⌧ :⌧ +n )|V (S⌧ )
V (S⌧ ) + ↵ [G V (S⌧ )]Until ⌧ = T 1

place of the error in (7.2) would actually be a slightly di↵erent algorithm. Would it be abetter algorithm or a worse one? Devise and program a small experiment to answer thisquestion empirically.
⇤
The n-step return uses the value function Vt+n 1 to correct for the missing rewards
beyond Rt+n . An important property of n-step returns is that their expectation is
guaranteed to be a better estimate of v⇡ than Vt+n 1 is, in a worst-state sense. That is,
the worst error of the expected n-step return is guaranteed to be less than or equal to ntimes the worst error under Vt+n 1 :max E⇡[Gt:t+n |St = s]sv⇡ (s) nmax Vt+n 1 (s)sv⇡ (s) ,(7.3)

for all n 1. This is called the error reduction property of n-step returns. Because of theerror reduction property, one can show formally that all n-step TD methods converge tothe correct predictions under appropriate technical conditions. The n-step TD methods
thus form a family of sound methods, with one-step TD methods and Monte Carlo
methods as extreme members.
Example 7.1: n-step TD Methods on the Random Walk Consider using n-step
TD methods on the 5-state random walk task described in Example 6.2 (page 125).
Suppose the first episode progressed directly from the center state, C, to the right,
through D and E, and then terminated on the right with a return of 1. Recall that theestimated values of all the states started at an intermediate value, V (s) = 0.5. As a resultof this experience, a one-step method would change only the estimate for the last state,7.2. n-step Sarsa

145

V (E), which would be incremented toward 1, the observed return. A two-step method,on the other hand, would increment the values of the two states preceding termination:
V (D) and V (E) both would be incremented toward 1. A three-step method, or any n-stepmethod for n > 2, would increment the values of all three of the visited states toward 1,
all by the same amount.
Which value of n is better? Figure 7.2 shows the results of a simple empirical test fora larger random walk process, with 19 states instead of 5 (and with a 1 outcome on theleft, all values initialized to 0), which we use as a running example in this chapter. Results
are shown for n-step TD methods with a range of values for n and ↵. The performancemeasure for each parameter setting, shown on the vertical axis, is the square-root of
the average squared error between the predictions at the end of the episode for the 19
states and their true values, then averaged over the first 10 episodes and 100 repetitionsof the whole experiment (the same sets of walks were used for all parameter settings).
Note that methods with an intermediate value of n worked best. This illustrates how
the generalization of TD and Monte Carlo methods to n-step methods can potentiallyperform better than either of the two extreme methods.
256
512

0.55

128

n=64

n=32

0.5

Average
RMS error
over 19 statesand first 10
episodes

0.45
0.4
0.35

n=32

0.3

n=16

n=1

n=8
0.25
0

0.2

n=2

n=4
0.4

0.6

0.8

1

↵
Figure 7.2: Performance of n-step TD methods as a function of ↵, for various values of n, on
a 19-state random walk task (Example 7.1).

Exercise 7.3 Why do you think a larger random walk task (19 states instead of 5) wasused in the examples of this chapter? Would a smaller walk have shifted the advantage
to a di↵erent value of n? How about the change in left-side outcome from 0 to 1 madein the larger walk? Do you think that made any di↵erence in the best value of n?
⇤

7.2

n-step Sarsa

How can n-step methods be used not just for prediction, but for control? In this section
we show how n-step methods can be combined with Sarsa in a straightforward way to

146

Chapter 7: n-step Bootstrapping

produce an on-policy TD control method. The n-step version of Sarsa we call n-stepSarsa, and the original version presented in the previous chapter we henceforth call
one-step Sarsa, or Sarsa(0).
The main idea is to simply switch states for actions (state–action pairs) and then usean "-greedy policy. The backup diagrams for n-step Sarsa (shown in Figure 7.3), likethose of n-step TD (Figure 7.1), are strings of alternating states and actions, except that
the Sarsa ones all start and end with an action rather a state. We redefine n-step returns(update targets) in terms of estimated action values:.
Gt:t+n = Rt+1 + Rt+2 +· · ·+ n 1 Rt+n + n Qt+n 1 (St+n , At+n ), n.
with Gt:t+n = Gt if t + n

1, 0  t < T

n,

(7.4)

T . The natural algorithm is then.
Qt+n (St , At ) = Qt+n 1 (St , At ) + ↵ [Gt:t+n

Qt+n 1 (St , At )] ,0  t < T, (7.5)

while the values of all other states remain unchanged: Qt+n (s, a) = Qt+n 1 (s, a), for alls, a such that s 6= St or a 6= At . This is the algorithm we call n-step Sarsa. Pseudocode
is shown in the box on the next page, and an example of why it can speed up learningcompared to one-step methods is given in Figure 7.4.
1-step Sarsa
aka Sarsa(0)2-step Sarsa

3-step Sarsa

n-step Sarsa∞-step Sarsa
aka Monte Carlon-stepExpected Sarsa

Figure 7.3: The backup diagrams for the spectrum of n-step methods for state–action values.
They range from the one-step update of Sarsa(0) to the up-until-termination update of theMonte Carlo method. In between are the n-step updates, based on n steps of real rewards andthe estimated value of the nth next state–action pair, all appropriately discounted. On the far
right is the backup diagram for n-step Expected Sarsa.

7.2. n-step Sarsa

147

n-step Sarsa for estimating Q ⇡ q⇤ or q⇡
Initialize Q(s, a) arbitrarily, for all s 2 S, a 2 A
Initialize ⇡ to be "-greedy with respect to Q, or to a fixed given policy
Algorithm parameters: step size ↵ 2 (0, 1], small " > 0, a positive integer n
All store and access operations (for St , At , and Rt ) can take their index mod n + 1
Loop for each episode:
Initialize and store S0 6= terminal
Select and store an action A0 ⇠ ⇡(·|S0 )T
1Loop for t = 0, 1, 2, . . . :| If t < T , then:|Take action At|
Observe and store the next reward as Rt+1 and the next state as St+1|
If St+1 is terminal, then:
|
T
t+1
|
else:
|
Select and store an action At+1 ⇠ ⇡(·|St+1 )| ⌧t n + 1 (⌧ is the time whose estimate is being updated)| If ⌧ 0:
Pmin(⌧ +n,T ) i ⌧ 1|
GRi
i=⌧ +1|
If ⌧ + n < T , then G
G + n Q(S⌧ +n , A⌧ +n )(G⌧ :⌧ +n )|Q(S⌧ , A⌧ )Q(S⌧ , A⌧ ) + ↵ [G Q(S⌧ , A⌧ )]|If ⇡ is being learned, then ensure that ⇡(·|S⌧ ) is "-greedy wrt Q
Until ⌧ = T 1

Path taken

G

Action values increased
by one-step Sarsa

G

Action values increasedby 10-step Sarsa

G

Figure 7.4: Gridworld example of the speedup of policy learning due to the use of n-stepmethods. The first panel shows the path taken by an agent in a single episode, ending at alocation of high reward, marked by the G. In this example the values were all initially 0, and all
rewards were zero except for a positive reward at G. The arrows in the other two panels show
which action values were strengthened as a result of this path by one-step and n-step Sarsa
methods. The one-step method strengthens only the last action of the sequence of actions that
led to the high reward, whereas the n-step method strengthens the last n actions of the sequence,
so that much more is learned from the one episode.

148

Chapter 7: n-step Bootstrapping

Exercise 7.4 Prove that the n-step return of Sarsa (7.4) can be written exactly in termsof a novel TD error, asmin(t+n,T ) 1

Gt:t+n = Qt 1 (St , At ) +X

k t[Rk+1 + Qk (Sk+1 , Ak+1 )Qk 1 (Sk , Ak )] .

k=t

(7.6)

⇤
What about Expected Sarsa? The backup diagram for the n-step version of Expected
Sarsa is shown on the far right in Figure 7.3. It consists of a linear string of sample
actions and states, just as in n-step Sarsa, except that its last element is a branch over
all action possibilities weighted, as always, by their probability under ⇡. This algorithm
can be described by the same equation as n-step Sarsa (above) except with the n-stepreturn redefined as.
Gt:t+n = Rt+1 + · · · + n 1 Rt+n + n V̄t+n 1 (St+n ),t + n < T,
(7.7).
(with Gt:t+n = Gt for t + n T ) where V̄t (s) is the expected approximate value of state s,using the estimated action values at time t, under the target policy:. X
V̄t (s) =⇡(a|s)Qt (s, a),for all s 2 S.
(7.8)
a

Expected approximate values are used in developing many of the action-value methods
in the rest of this book. If s is terminal, then its expected approximate value is defined
to be 0.

7.3

n-step O↵-policy Learning

Recall that o↵-policy learning is learning the value function for one policy, ⇡, while
following another policy, b. Often, ⇡ is the greedy policy for the current action-valuefunction estimate, and b is a more exploratory policy, perhaps "-greedy. In order to
use the data from b we must take into account the di↵erence between the two policies,
using their relative probability of taking the actions that were taken (see Section 5.5). In
n-step methods, returns are constructed over n steps, so we are interested in the relative
probability of just those n actions. For example, to make a simple o↵-policy version ofn-step TD, the update for time t (actually made at time t + n) can simply be weighted
by ⇢t:t+n 1 :.
Vt+n (St ) = Vt+n 1 (St ) + ↵⇢t:t+n 1 [Gt:t+n Vt+n 1 (St )] ,0  t < T,
(7.9)
where ⇢t:t+n 1 , called the importance sampling ratio, is the relative probability under
the two policies of taking the n actions from At to At+n 1 (cf. Eq. 5.3):.
⇢t:h =min(h,T

Y

k=t

1)

⇡(Ak |Sk ).
b(Ak |Sk )(7.10)

7.3. n-step O↵-policy Learning

149

For example, if any one of the actions would never be taken by ⇡ (i.e., ⇡(Ak |Sk ) = 0) then
the n-step return should be given zero weight and be totally ignored. On the other hand,
if by chance an action is taken that ⇡ would take with much greater probability than bdoes, then this will increase the weight that would otherwise be given to the return. This
makes sense because that action is characteristic of ⇡ (and therefore we want to learn
about it) but is selected only rarely by b and thus rarely appears in the data. To makeup for this we have to over-weight it when it does occur. Note that if the two policies
are actually the same (the on-policy case) then the importance sampling ratio is always
1. Thus our new update (7.9) generalizes and can completely replace our earlier n-step
TD update. Similarly, our previous n-step Sarsa update can be completely replaced by asimple o↵-policy form:.
Qt+n (St , At ) = Qt+n 1 (St , At ) + ↵⇢t+1:t+n [Gt:t+n Qt+n 1 (St , At )] ,(7.11)
for 0  t < T . Note that the importance sampling ratio here starts and ends one step
later than for n-step TD (7.9). This is because here we are updating a state–action
pair. We do not have to care how likely we were to select the action; now that we haveselected it we want to learn fully from what happens, with importance sampling only for
subsequent actions. Pseudocode for the full algorithm is shown in the box below.
O↵-policy n-step Sarsa for estimating Q ⇡ q⇤ or q⇡
Input: an arbitrary behavior policy b such that b(a|s) > 0, for all s 2 S, a 2 A
Initialize Q(s, a) arbitrarily, for all s 2 S, a 2 A
Initialize ⇡ to be greedy with respect to Q, or as a fixed given policy
Algorithm parameters: step size ↵ 2 (0, 1], a positive integer n
All store and access operations (for St , At , and Rt ) can take their index mod n + 1
Loop for each episode:
Initialize and store S0 6= terminal
Select and store an action A0 ⇠ b(·|S0 )T
1Loop for t = 0, 1, 2, . . . :| If t < T , then:|Take action At|
Observe and store the next reward as Rt+1 and the next state as St+1|
If St+1 is terminal, then:
|
T
t+1
|
else:
|
Select and store an action At+1 ⇠ b(·|St+1 )| ⌧t n + 1 (⌧ is the time whose estimate is being updated)| If ⌧ 0:
Qmin(⌧ +n,T 1) ⇡(Ai |Si )(⇢⌧ +1:⌧ +n )|⇢+1b(Ai |Si )Pi=⌧min(⌧ +n,T ) i ⌧ 1|
GRii=⌧ +1|
If ⌧ + n < T , then: G
G + n Q(S⌧ +n , A⌧ +n )(G⌧ :⌧ +n )|Q(S⌧ , A⌧ )Q(S⌧ , A⌧ ) + ↵⇢ [G Q(S⌧ , A⌧ )]|If ⇡ is being learned, then ensure that ⇡(·|S⌧ ) is greedy wrt Q
Until ⌧ = T 1

150

Chapter 7: n-step Bootstrapping

The o↵-policy version of n-step Expected Sarsa would use the same update as above
for n-step Sarsa except that the importance sampling ratio would have one less factor in
it. That is, the above equation would use ⇢t+1:t+n 1 instead of ⇢t+1:t+n , and of course
it would use the Expected Sarsa version of the n-step return (7.7). This is because in
Expected Sarsa all possible actions are taken into account in the last state; the one
actually taken has no e↵ect and does not have to be corrected for.

7.4

*Per-decision Methods with Control Variates

The multi-step o↵-policy methods presented in the previous section are simple andconceptually clear, but are probably not the most efficient. A more sophisticated approach
would use per-decision importance sampling ideas such as were introduced in Section 5.9.
To understand this approach, first note that the ordinary n-step return (7.1), like all
returns, can be written recursively. For the n steps ending at horizon h, the n-step returncan be written
Gt:h = Rt+1 + Gt+1:h ,t < h < T,

(7.12).
where Gh:h = Vh 1 (Sh ). (Recall that this return is used at time h, previously denotedt + n.) Now consider the e↵ect of following a behavior policy b that is not the sameas the target policy ⇡. All of the resulting experience, including the first reward Rt+1
and the next state St+1 , must be weighted by the importance sampling ratio for time t,t |St )
⇢t = ⇡(Ab(At |St ) . One might be tempted to simply weight the righthand side of the above
equation, but one can do better. Suppose the action at time t would never be selected by⇡, so that ⇢t is zero. Then a simple weighting would result in the n-step return being
zero, which could result in high variance when it was used as a target. Instead, in thismore sophisticated approach, one uses an alternate, o↵-policy definition of the n-stepreturn ending at horizon h, as.
Gt:h = ⇢t (Rt+1 + Gt+1:h ) + (1⇢t )Vh 1 (St ),t < h < T,

(7.13).
where again Gh:h = Vh 1 (Sh ). In this approach, if ⇢t is zero, then instead of the target
being zero and causing the estimate to shrink, the target is the same as the estimate andcauses no change. The importance sampling ratio being zero means we should ignore thesample, so leaving the estimate unchanged seems appropriate. The second, additionalterm in (7.13) is called a control variate (for obscure reasons). Notice that the control
variate does not change the expected update; the importance sampling ratio has expectedvalue one (Section 5.9) and is uncorrelated with the estimate, so the expected valueof the control variate is zero. Also note that the o↵-policy definition (7.13) is a strict
generalization of the earlier on-policy definition of the n-step return (7.1), as the two are
identical in the on-policy case, in which ⇢t is always 1.
For a conventional n-step method, the learning rule to use in conjunction with (7.13)
is the n-step TD update (7.2), which has no explicit importance sampling ratios otherthan those embedded in the return.
Exercise 7.5 Write the pseudocode for the o↵-policy state-value prediction algorithm
described above.
⇤7.4. Per-decision Methods with Control Variates151

For action values, the o↵-policy definition of the n-step return is a little di↵erent
because the first action does not play a role in the importance sampling. That first actionis the one being learned; it does not matter if it was unlikely or even impossible under thetarget policy—it has been taken and now full unit weight must be given to the reward
and state that follows it. Importance sampling will apply only to the actions that follow
it.
First note that for action values the n-step on-policy return ending at horizon h,
expectation form (7.7), can be written recursively just as in (7.12), except that for action.
values the recursion ends with Gh:h = V̄h 1 (Sh ) as in (7.8). An o↵-policy form with
control variates is⇣⌘.
Gt:h = Rt+1 + ⇢t+1 Gt+1:h + V̄h 1 (St+1 ) ⇢t+1 Qh 1 (St+1 , At+1 ) ,⇣⌘= Rt+1 + ⇢t+1 Gt+1:h Qh 1 (St+1 , At+1 ) + V̄h 1 (St+1 ), t < h  T.
(7.14).
If h < T , then the recursion ends with Gh:h = Qh 1 (Sh , Ah ), whereas, if hT,.
the recursion ends with and GT 1:h = RT . The resultant prediction algorithm (after
combining with (7.5)) is analogous to Expected Sarsa.
Exercise 7.6 Prove that the control variate in the above equations does not change theexpected value of the return.
⇤⇤

Exercise 7.7 Write the pseudocode for the o↵-policy action-value prediction algorithm
described immediately above. Pay particular attention to the termination conditions forthe recursion upon hitting the horizon or the end of episode.
⇤
Exercise 7.8 Show that the general (o↵-policy) version of the n-step return (7.13) canstill be written exactly and compactly as the sum of state-based TD errors (6.5) if theapproximate state value function does not change.
⇤
Exercise 7.9 Repeat the above exercise for the action version of the o↵-policy n-stepreturn (7.14) and the Expected Sarsa TD error (the quantity in brackets in Equation 6.9).
⇤
Exercise 7.10 (programming) Devise a small o↵-policy prediction problem and use it toshow that the o↵-policy learning algorithm using (7.13) and (7.2) is more data efficientthan the simpler algorithm using (7.1) and (7.9).
⇤
The importance sampling that we have used in this section, the previous section, andin Chapter 5, enables sound o↵-policy learning, but also results in high variance updates,
forcing the use of a small step-size parameter and thereby causing learning to be slow. It
is probably inevitable that o↵-policy training is slower than on-policy training—after all,the data is less relevant to what is being learned. However, it is probably also true thatthese methods can be improved on. The control variates are one way of reducing thevariance. Another is to rapidly adapt the step sizes to the observed variance, as in the
Autostep method (Mahmood, Sutton, Degris and Pilarski, 2012). Yet another promising
approach is the invariant updates of Karampatziakis and Langford (2010) as extended
to TD by Tian (in preparation). The usage technique of Mahmood (2017; Mahmood

152

Chapter 7: n-step Bootstrappingand Sutton, 2015) may also be part of the solution. In the next section we consider ano↵-policy learning method that does not use importance sampling.

7.5O↵-policy Learning Without Importance Sampling:
The n-step Tree Backup Algorithm

Is o↵-policy learning possible without importance sampling? Q-learning and Expected
Sarsa from Chapter 6 do this for the one-step case, but is there a corresponding multi-stepalgorithm? In this section we present just such an n-step method, called the tree-backup
algorithm.
The idea of the algorithm is suggested by the 3-step tree-backup backup
St , At
diagram shown to the right. Down the central spine and labeled in the
diagram are three sample states and rewards, and two sample actions. Rt+1
These are the random variables representing the events occurring after theSt+1
initial state–action pair St , At . Hanging o↵ to the sides of each state are
the actions that were not selected. (For the last state, all the actions are
At+1
considered to have not (yet) been selected.) Because we have no sample
Rt+2
data for the unselected actions, we bootstrap and use the estimates of
St+2
their values in forming the target for the update. This slightly extends theidea of a backup diagram. So far we have always updated the estimatedvalue of the node at the top of the diagram toward a target combining
At+2
Rt+3
the rewards along the way (appropriately discounted) and the estimatedvalues of the nodes at the bottom. In the tree-backup update, the target
St+3
includes all these things plus the estimated values of the dangling action
nodes hanging o↵ the sides, at all levels. This is why it is called a treebackup update; it is an update from the entire tree of estimated action
the 3-stepvalues.
tree-backup
updateMore precisely, the update is from the estimated action values of theleaf nodes of the tree. The action nodes in the interior, corresponding to
the actual actions taken, do not participate. Each leaf node contributes to the target
with a weight proportional to its probability of occurring under the target policy ⇡. Thus
each first-level action a contributes with a weight of ⇡(a|St+1 ), except that the action
actually taken, At+1 , does not contribute at all. Its probability, ⇡(At+1 |St+1 ), is used
to weight all the second-level action values. Thus, each non-selected second-level action
a0 contributes with weight ⇡(At+1 |St+1 )⇡(a0 |St+2 ). Each third-level action contributeswith weight ⇡(At+1 |St+1 )⇡(At+2 |St+2 )⇡(a00 |St+3 ), and so on. It is as if each arrow to an
action node in the diagram is weighted by the action’s probability of being selected under
the target policy and, if there is a tree below the action, then that weight applies to all
the leaf nodes in the tree.

7.5. O↵-policy Learning Without Importance Sampling: n-step Tree Backup

153

We can think of the 3-step tree-backup update as consisting of 6 half-steps, alternating
between sample half-steps from an action to a subsequent state, and expected half-steps
considering from that state all possible actions with their probabilities of occurring under
the policy.
Now let us develop the detailed equations for the n-step tree-backup algorithm. The
one-step return (target) is the same as that of Expected Sarsa,
X.
Gt:t+1 = Rt+1 +⇡(a|St+1 )Qt (St+1 , a),(7.15)
afor t < T

1, and the two-step tree-backup return isX.
Gt:t+2 = Rt+1 +⇡(a|St+1 )Qt+1 (St+1 , a)a6=At+1= Rt+1 +⇣⌘X+ ⇡(At+1 |St+1 ) Rt+2 +⇡(a|St+2 )Qt+1 (St+2 , a)X

a⇡(a|St+1 )Qt+1 (St+1 , a) + ⇡(At+1 |St+1 )Gt+1:t+2 ,a6=At+1

for t < T 2. The latter form suggests the general recursive definition of the tree-backup
n-step return:X.
Gt:t+n = Rt+1 +⇡(a|St+1 )Qt+n 1 (St+1 , a) + ⇡(At+1 |St+1 )Gt+1:t+n , (7.16)
a6=At+1.
for t < T 1, n 2, with the n = 1 case handled by (7.15) except for GT 1:t+n = RT .
This target is then used with the usual action-value update rule from n-step Sarsa:.
Qt+n (St , At ) = Qt+n 1 (St , At ) + ↵ [Gt:t+n

Qt+n 1 (St , At )] ,for 0  t < T , while the values of all other state–action pairs remain unchanged:
Qt+n (s, a) = Qt+n 1 (s, a), for all s, a such that s 6= St or a 6= At . Pseudocode for thisalgorithm is shown in the box on the next page.
Exercise 7.11 Show that if the approximate action values are unchanging, then thetree-backup return (7.16) can be written as a sum of expectation-based TD errors:min(t+n 1,T

Gt:t+n = Q(St , At ) +

X
k=t

where

.

t = Rt+1 +V̄t (St+1 )

1)
k

k
Y

i=t+1

⇡(Ai |Si ),Q(St , At ) and V̄t is given by (7.8).

⇤

154

Chapter 7: n-step Bootstrappingn-step Tree Backup for estimating Q ⇡ q⇤ or q⇡
Initialize Q(s, a) arbitrarily, for all s 2 S, a 2 A
Initialize ⇡ to be greedy with respect to Q, or as a fixed given policy
Algorithm parameters: step size ↵ 2 (0, 1], a positive integer n
All store and access operations can take their index mod n + 1
Loop for each episode:
Initialize and store S0 6= terminal
Choose an action A0 arbitrarily as a function of S0 ; Store A0
T
1
Loop for t = 0, 1, 2, . . . :| If t < T :|Take action At ; observe and store the next reward and state as Rt+1 , St+1|
If St+1 is terminal:
|
T
t+1
|
else:
|
Choose an action At+1 arbitrarily as a function of St+1 ; Store At+1
| ⌧t + 1 n (⌧ is the time whose estimate is being updated)| If ⌧ 0:|
If t + 1 T :
|
G
RT
|
else
P
|
G
Rt+1 +a ⇡(a|St+1 )Q(St+1 , a)|Loop for k = min(t,P T 1) down through ⌧ + 1:|GRk +a6=Ak ⇡(a|Sk )Q(Sk , a) + ⇡(Ak |Sk )G|
Q(S⌧ , A⌧ )Q(S⌧ , A⌧ ) + ↵ [G Q(S⌧ , A⌧ )]|If ⇡ is being learned, then ensure that ⇡(·|S⌧ ) is greedy wrt Q
Until ⌧ = T 1

7.6*A Unifying Algorithm: n-step Q( )So far in this chapter we have considered three di↵erent kinds of action-value algorithms,
corresponding to the first three backup diagrams shown in Figure 7.5. n-step Sarsa has
all sample transitions, the tree-backup algorithm has all state-to-action transitions fully
branched without sampling, and n-step Expected Sarsa has all sample transitions except
for the last state-to-action one, which is fully branched with an expected value. To what
extent can these algorithms be unified?
One idea for unification is suggested by the fourth backup diagram in Figure 7.5. Thisis the idea that one might decide on a step-by-step basis whether one wanted to take theaction as a sample, as in Sarsa, or consider the expectation over all actions instead, as in
the tree-backup update. Then, if one chose always to sample, one would obtain Sarsa,
whereas if one chose never to sample, one would get the tree-backup algorithm. Expected
Sarsa would be the case where one chose to sample for all steps except for the last one.

7.6. A Unifying Algorithm: n-step Q( )155

4-stepExpected Sarsa

4-step

⇢

⇢

⇢

⇢

⇢

⇢

⇢

4-stepSarsa

4-stepTree backup

Q( )

=1

=0

=1
⇢

=0
⇢

Figure 7.5: The backup diagrams of the three kinds of n-step action-value updates considered
so far in this chapter (4-step case) plus the backup diagram of a fourth kind of update that unifies
them all. The label ‘⇢’ indicates half transitions on which importance sampling is required in theo↵-policy case. The fourth kind of update unifies all the others by choosing on a state-by-statebasis whether to sample ( t = 1) or not ( t = 0).

And of course there would be many other possibilities, as suggested by the last diagramin the figure. To increase the possibilities even further we can consider a continuous
variation between sampling and expectation. Let t 2 [0, 1] denote the degree of samplingon step t, with = 1 denoting full sampling and = 0 denoting a pure expectation withno sampling. The random variable t might be set as a function of the state, action, orstate–action pair at time t. We call this proposed new algorithm n-step Q( ).
Now let us develop the equations of n-step Q( ). First we write the tree-backup
n-step return (7.16) in terms of the horizon h = t + n and then in terms of the expected
approximate value V̄ (7.8):
X
Gt:h = Rt+1 +⇡(a|St+1 )Qh 1 (St+1 , a) + ⇡(At+1 |St+1 )Gt+1:ha6=At+1⇡(At+1 |St+1 )Qh 1 (St+1 , At+1 ) + ⇡(At+1 |St+1 )Gt+1:h⌘= Rt+1 + ⇡(At+1 |St+1 ) Gt+1:h Qh 1 (St+1 , At+1 ) + V̄h 1 (St+1 ),= Rt+1 + V̄h 1 (St+1 )⇣after which it is exactly like the n-step return for Sarsa with control variates (7.14) except
with the action probability ⇡(At+1 |St+1 ) substituted for the importance-sampling ratio⇢t+1 . For Q( ), we slide linearly between these two cases:⇣⌘⇣⌘.
Gt:h = Rt+1 +
⇢
+
(1
)⇡(A
|S
)
G
Q
(S
,
A
)
t+1 t+1
t+1
t+1 t+1
t+1:h
h 1
t+1
t+1
+ V̄h 1 (St+1 ),(7.17)

156

Chapter 7: n-step Bootstrapping.
for t < h  T . The recursion ends with Gh:h = Qh 1 (Sh , Ah ) if h < T , or with.
GT 1:T = RT if h = T . Then we use the earlier update for n-step Sarsa without
importance-sampling ratios (7.5) instead of (7.11), because now the ratios are incorporated
in the n-step return. A complete algorithm is given in the box.
O↵-policy n-step Q( ) for estimating Q ⇡ q⇤ or q⇡
Input: an arbitrary behavior policy b such that b(a|s) > 0, for all s 2 S, a 2 A
Initialize Q(s, a) arbitrarily, for all s 2 S, a 2 A
Initialize ⇡ to be greedy with respect to Q, or else it is a fixed given policy
Algorithm parameters: step size ↵ 2 (0, 1], a positive integer n
All store and access operations can take their index mod n + 1
Loop for each episode:
Initialize and store S0 6= terminal
Choose and store an action A0 ⇠ b(·|S0 )T
1Loop for t = 0, 1, 2, . . . :| If t < T :|Take action At ; observe and store the next reward and state as Rt+1 , St+1|
If St+1 is terminal:
|
T
t+1
|
else:
|
Choose and store an action At+1 ⇠ b(·|St+1 )|Select and store t+1
t+1 |St+1 )|Store ⇡(Ab(At+1 |St+1 ) as ⇢t+1
| ⌧t n + 1 (⌧ is the time whose estimate is being updated)| If ⌧ 0:|
If t + 1 < T :|G
Q(St+1 , At+1 )|Loop for k = min(t + 1, T ) down through ⌧ + 1:|
if k = T :
|
G
RT
|
else:
P
|
V̄
a ⇡(a|Sk )Q(Sk , a)|G
Rk +Q(Sk , Ak ) + V̄k ⇢k + (1k )⇡(Ak |Sk ) G|
Q(S⌧ , A⌧ )Q(S⌧ , A⌧ ) + ↵ [G Q(S⌧ , A⌧ )]|If ⇡ is being learned, then ensure that ⇡(·|S⌧ ) is greedy wrt Q
Until ⌧ = T 1

7.7. Summary

7.7

157

Summary

In this chapter we have developed a range of temporal-di↵erence learning methods that lie
in between the one-step TD methods of the previous chapter and the Monte Carlo methodsof the chapter before. Methods that involve an intermediate amount of bootstrapping
are important because they will typically perform better than either extreme.
Our focus in this chapter has been on n-step methods, whichlook ahead to the next n rewards, states, and actions. The two
4-step backup diagrams to the right together summarize most of the⇢methods introduced. The state-value update shown is for n-step=1⇢
TD with importance sampling, and the action-value update is forn-step Q( ), which generalizes Expected Sarsa and Q-learning. All⇢n-step methods involve a delay of n time steps before updating,=0
as only then are all the required future events known. A further
drawback is that they involve more computation per time stepthan previous methods. Compared to one-step methods, n-step⇢methods also require more memory to record the states, actions,=1
rewards, and sometimes other variables over the last n time steps.
⇢
Eventually, in Chapter 12, we will see how multi-step TD methods
can be implemented with minimal memory and computational⇢
complexity using eligibility traces, but there will always be some=0
additional computation beyond one-step methods. Such costs canbe well worth paying to escape the tyranny of the single time step.
Although n-step methods are more complex than those using
4-stepeligibility traces, they have the great benefit of being conceptually 4-stepQ( )TDclear. We have sought to take advantage of this by developing two
approaches to o↵-policy learning in the n-step case. One, based on
importance sampling is conceptually simple but can be of high variance. If the target andbehavior policies are very di↵erent it probably needs some new algorithmic ideas before
it can be efficient and practical. The other, based on tree-backup updates, is the naturalextension of Q-learning to the multi-step case with stochastic target policies. It involves
no importance sampling but, again if the target and behavior policies are substantiallydi↵erent, the bootstrapping may span only a few steps even if n is large.

158

Chapter 7: n-step Bootstrapping

Bibliographical and Historical Remarks
The notion of n-step returns is due to Watkins (1989), who also first discussed their error
reduction property. n-step algorithms were explored in the first edition of this book,in which they were treated as of conceptual interest, but not feasible in practice. The
work of Cichosz (1995) and particularly van Seijen (2016) showed that they are actuallycompletely practical algorithms. Given this, and their conceptual clarity and simplicity,
we have chosen to highlight them here in the second edition. In particular, we now
postpone all discussion of the backward view and of eligibility traces until Chapter 12.
7.1–2 The results in the random walk examples were made for this text based on workof Sutton (1988) and Singh and Sutton (1996). The use of backup diagrams todescribe these and other algorithms in this chapter is new.
7.3–5 The developments in these sections are based on the work of Precup, Sutton,
and Singh (2000), Precup, Sutton, and Dasgupta (2001), and Sutton, Mahmood,
Precup, and van Hasselt (2014).
The tree-backup algorithm is due to Precup, Sutton, and Singh (2000), but thepresentation of it here is new.
7.6

The Q( ) algorithm is new to this text, but closely related algorithms have been
explored further by De Asis, Hernandez-Garcia, Holland, and Sutton (2017).

Chapter 8

Planning and Learning with
Tabular Methods
In this chapter we develop a unified view of reinforcement learning methods that requirea model of the environment, such as dynamic programming and heuristic search, andmethods that can be used without a model, such as Monte Carlo and temporal-di↵erence
methods. These are respectively called model-based and model-free reinforcement learning
methods. Model-based methods rely on planning as their primary component, whilemodel-free methods primarily rely on learning. Although there are real di↵erences between
these two kinds of methods, there are also great similarities. In particular, the heart ofboth kinds of methods is the computation of value functions. Moreover, all the methods
are based on looking ahead to future events, computing a backed-up value, and thenusing it as an update target for an approximate value function. Earlier in this book we
presented Monte Carlo and temporal-di↵erence methods as distinct alternatives, thenshowed how they can be unified by n-step methods. Our goal in this chapter is a similar
integration of model-based and model-free methods. Having established these as distinct
in earlier chapters, we now explore the extent to which they can be intermixed.

8.1

Models and Planning

By a model of the environment we mean anything that an agent can use to predict how theenvironment will respond to its actions. Given a state and an action, a model produces aprediction of the resultant next state and next reward. If the model is stochastic, thenthere are several possible next states and next rewards, each with some probability ofoccurring. Some models produce a description of all possibilities and their probabilities;
these we call distribution models. Other models produce just one of the possibilities,
sampled according to the probabilities; these we call sample models. For example, consider
modeling the sum of a dozen dice. A distribution model would produce all possible sums
and their probabilities of occurring, whereas a sample model would produce an individual

160

Chapter 8: Planning and Learning with Tabular Methods

sum drawn according to this probability distribution. The kind of model assumed indynamic programming—estimates of the MDP’s dynamics, p(s0 , r |s, a)—is a distribution
model. The kind of model used in the blackjack example in Chapter 5 is a sample model.
Distribution models are stronger than sample models in that they can always be used
to produce samples. However, in many applications it is much easier to obtain sample
models than distribution models. The dozen dice are a simple example of this. It would
be easy to write a computer program to simulate the dice rolls and return the sum, butharder and more error-prone to figure out all the possible sums and their probabilities.
Models can be used to mimic or simulate experience. Given a starting state and action,a sample model produces a possible transition, and a distribution model generates all
possible transitions weighted by their probabilities of occurring. Given a starting state
and a policy, a sample model could produce an entire episode, and a distribution model
could generate all possible episodes and their probabilities. In either case, we say themodel is used to simulate the environment and produce simulated experience.
The word planning is used in several di↵erent ways in di↵erent fields. We use theterm to refer to any computational process that takes a model as input and produces or
improves a policy for interacting with the modeled environment:

model

planning

policyIn artificial intelligence, there are two distinct approaches to planning according to ourdefinition. State-space planning, which includes the approach we take in this book,is viewed primarily as a search through the state space for an optimal policy or an
optimal path to a goal. Actions cause transitions from state to state, and value functions
are computed over states. In what we call plan-space planning, planning is instead asearch through the space of plans. Operators transform one plan into another, andvalue functions, if any, are defined over the space of plans. Plan-space planning includes
evolutionary methods and “partial-order planning,” a common kind of planning in artificial
intelligence in which the ordering of steps is not completely determined at all stages ofplanning. Plan-space methods are difficult to apply efficiently to the stochastic sequential
decision problems that are the focus in reinforcement learning, and we do not consider
them further (but see, e.g., Russell and Norvig, 2010).
The unified view we present in this chapter is that all state-space planning methods
share a common structure, a structure that is also present in the learning methods
presented in this book. It takes the rest of the chapter to develop this view, but there are
two basic ideas: (1) all state-space planning methods involve computing value functions
as a key intermediate step toward improving the policy, and (2) they compute valuefunctions by updates or backup operations applied to simulated experience. This common
structure can be diagrammed as follows:model

simulated
experience

updates
backups
backupsvaluespolicyDynamic programming methods clearly fit this structure: they make sweeps through thespace of states, generating for each state the distribution of possible transitions. Each
distribution is then used to compute a backed-up value (update target) and update the8.2. Dyna: Integrated Planning, Acting, and Learning

161

state’s estimated value. In this chapter we argue that various other state-space planning
methods also fit this structure, with individual methods di↵ering only in the kinds of
updates they do, the order in which they do them, and in how long the backed-up
information is retained.
Viewing planning methods in this way emphasizes their relationship to the learning
methods that we have described in this book. The heart of both learning and planning
methods is the estimation of value functions by backing-up update operations. The
di↵erence is that whereas planning uses simulated experience generated by a model,
learning methods use real experience generated by the environment. Of course thisdi↵erence leads to a number of other di↵erences, for example, in how performance isassessed and in how flexibly experience can be generated. But the common structuremeans that many ideas and algorithms can be transferred between planning and learning.
In particular, in many cases a learning algorithm can be substituted for the key update
step of a planning method. Learning methods require only experience as input, and inmany cases they can be applied to simulated experience just as well as to real experience.
The box below shows a simple example of a planning method based on one-step tabular
Q-learning and on random samples from a sample model. This method, which we callrandom-sample one-step tabular Q-planning, converges to the optimal policy for the model
under the same conditions that one-step tabular Q-learning converges to the optimal
policy for the real environment (each state–action pair must be selected an infinite numberof times in Step 1, and ↵ must decrease appropriately over time).
Random-sample one-step tabular Q-planning
Loop forever:
1. Select a state, S 2 S, and an action, A 2 A(S), at random
2. Send S, A to a sample model, and obtaina sample next reward, R, and a sample next state, S 0
3. Apply one-step tabular Q-learning
to S, A, R, S 0 :⇥⇤Q(S, A)
Q(S, A) + ↵ R + maxa Q(S 0 , a) Q(S, A)
In addition to the unified view of planning and learning methods, a second theme inthis chapter is the benefits of planning in small, incremental steps. This enables planningto be interrupted or redirected at any time with little wasted computation, which appears
to be a key requirement for efficiently intermixing planning with acting and with learningof the model. Planning in very small steps may be the most efficient approach even on
pure planning problems if the problem is too large to be solved exactly.

8.2

Dyna: Integrated Planning, Acting, and Learning

When planning is done online, while interacting with the environment, a number ofinteresting issues arise. New information gained from the interaction may change themodel and thereby interact with planning. It may be desirable to customize the planning
process in some way to the states or decisions currently under consideration, or expected

162

Chapter 8: Planning and Learning with Tabular Methods

in the near future. If decision making and model learning are both computation-intensive
processes, then the available computational resources may need to be divided between
them. To begin exploring these issues, in this section we present Dyna-Q, a simple
architecture integrating the major functions needed in an online planning agent. Eachfunction appears in Dyna-Q in a simple, almost trivial, form. In subsequent sections weelaborate some of the alternate ways of achieving each function and the trade-o↵s between
them. For now, we seek merely to illustrate the ideas and stimulate your intuition.
Within a planning agent, there are at least two roles for real experience: it can beused to improve the model (to make it more accurately match the real environment)
and it can be used to directly improve the value function and policy using the kinds ofreinforcement learning methods we have discussed
value/policy
in previous chapters. The former we call modellearning, and the latter we call direct reinforcement
actinglearning (direct RL). The possible relationships
planning
directbetween experience, model, values, and policy are
RL
summarized in the diagram to the right. Each arrow shows a relationship of influence and presumed
improvement. Note how experience can improveexperiencemodel
value functions and policies either directly or indirectly via the model. It is the latter, which ismodel
sometimes called indirect reinforcement learning,learningthat is involved in planning.
Both direct and indirect methods have advantages and disadvantages. Indirect methods
often make fuller use of a limited amount of experience and thus achieve a better policy
with fewer environmental interactions. On the other hand, direct methods are much
simpler and are not a↵ected by biases in the design of the model. Some have argued
that indirect methods are always superior to direct ones, while others have argued that
direct methods are responsible for most human and animal learning. Related debatesin psychology and artificial intelligence concern the relative importance of cognition as
opposed to trial-and-error learning, and of deliberative planning as opposed to reactive
decision making (see Chapter 14 for discussion of some of these issues from the perspectiveof psychology). Our view is that the contrast between the alternatives in all these debates
has been exaggerated, that more insight can be gained by recognizing the similarities
between these two sides than by opposing them. For example, in this book we haveemphasized the deep similarities between dynamic programming and temporal-di↵erence
methods, even though one was designed for planning and the other for model-free learning.
Dyna-Q includes all of the processes shown in the diagram above—planning, acting,model-learning, and direct RL—all occurring continually. The planning method is therandom-sample one-step tabular Q-planning method on page 161. The direct RL method
is one-step tabular Q-learning. The model-learning method is also table-based and assumes
the environment is deterministic. After each transition St , At ! Rt+1 , St+1 , the modelrecords in its table entry for St , At the prediction that Rt+1 , St+1 will deterministicallyfollow. Thus, if the model is queried with a state–action pair that has been experienced
before, it simply returns the last-observed next state and next reward as its prediction.

8.2. Dyna: Integrated Planning, Acting, and Learning

163

During planning, the Q-planning algorithm randomly samples only from state–action
pairs that have previously been experienced (in Step 1), so the model is never queried
with a pair about which it has no information.
The overall architecture of Dyna agents, of which the Dyna-Q algorithm is one example,
is shown in Figure 8.1. The central column represents the basic interaction between
agent and environment, giving rise to a trajectory of real experience. The arrow on theleft of the figure represents direct reinforcement learning operating on real experience toimprove the value function and the policy. On the right are model-based processes. The
model is learned from real experience and gives rise to simulated experience. We use theterm search control to refer to the process that selects the starting states and actions
for the simulated experiences generated by the model. Finally, planning is achieved byapplying reinforcement learning methods to the simulated experiences just as if they had
really happened. Typically, as in Dyna-Q, the same reinforcement learning method isused both for learning from real experience and for planning from simulated experience.
The reinforcement learning method is thus the “final common path” for both learningand planning. Learning and planning are deeply integrated in the sense that they sharealmost all the same machinery, di↵ering only in the source of their experience.

Policy/value functions
planning update
direct RL
updatesimulated
experience

realexperiencemodel
learning

searchcontrol

Model

Environment
Figure 8.1: The general Dyna Architecture. Real experience, passing back and forth between
the environment and the policy, a↵ects policy and value functions in much the same way as does
simulated experience generated by the model of the environment.

Conceptually, planning, acting, model-learning, and direct RL occur simultaneously
and in parallel in Dyna agents. For concreteness and implementation on a serial computer,however, we fully specify the order in which they occur within a time step. In Dyna-Q,
the acting, model-learning, and direct RL processes require little computation, and weassume they consume just a fraction of the time. The remaining time in each step can be
devoted to the planning process, which is inherently computation-intensive. Let us assume
that there is time in each step, after acting, model-learning, and direct RL, to complete

164

Chapter 8: Planning and Learning with Tabular Methods

n iterations (Steps 1–3) of the Q-planning algorithm. In the pseudocode algorithm for
Dyna-Q in the box below, M odel(s, a) denotes the contents of the model (predictednext state and reward) for state–action pair (s, a). Direct reinforcement learning, modellearning, and planning are implemented by steps (d), (e), and (f), respectively. If (e) and(f) were omitted, the remaining algorithm would be one-step tabular Q-learning.
Tabular Dyna-Q
Initialize Q(s, a) and M odel(s, a) for all s 2 S and a 2 A(s)Loop forever:(a) S
current (nonterminal) state(b) A"-greedy(S, Q)(c) Take action A; observe ⇥resultant reward, R, and state, ⇤S 0(d) Q(S, A)
Q(S, A) + ↵ R + maxa Q(S 0 , a) Q(S, A)(e) M odel(S, A)
R, S 0 (assuming deterministic environment)(f) Loop repeat n times:S
random previously observed stateA
random action previously taken in S
R, S 0M odel(S, A) ⇥⇤Q(S, A)
Q(S, A) + ↵ R + maxa Q(S 0 , a) Q(S, A)
Example 8.1: Dyna Maze Consider the simple maze shown inset in Figure 8.2. Ineach of the 47 states there are four actions, up, down, right, and left, which take theagent deterministically to the corresponding neighboring states, except when movementis blocked by an obstacle or the edge of the maze, in which case the agent remains where
it is. Reward is zero on all transitions, except those into the goal state, on which it is +1.
After reaching the goal state (G), the agent returns to the start state (S) to begin a new
episode. This is a discounted, episodic task with = 0.95.
The main part of Figure 8.2 shows average learning curves from an experiment inwhich Dyna-Q agents were applied to the maze task. The initial action values were zero,
the step-size parameter was ↵ = 0.1, and the exploration parameter was " = 0.1. Whenselecting greedily among actions, ties were broken randomly. The agents varied in thenumber of planning steps, n, they performed per real step. For each n, the curves show
the number of steps taken by the agent to reach the goal in each episode, averaged over 30
repetitions of the experiment. In each repetition, the initial seed for the random number
generator was held constant across algorithms. Because of this, the first episode wasexactly the same (about 1700 steps) for all values of n, and its data are not shown in
the figure. After the first episode, performance improved for all values of n, but much
more rapidly for larger values. Recall that the n = 0 agent is a nonplanning agent, using
only direct reinforcement learning (one-step tabular Q-learning). This was by far theslowest agent on this problem, despite the fact that the parameter values (↵ and ") were
optimized for it. The nonplanning agent took about 25 episodes to reach ("-)optimal
performance, whereas the n = 5 agent took about five episodes, and the n = 50 agent
took only three episodes.

8.2. Dyna: Integrated Planning, Acting, and Learning

165

G

800
S
actions

600

Steps
per
episode0 planning steps(direct RL only)400

5 planning steps
50 planning steps
200

14
2

10

20

30

40

50

EpisodesFigure 8.2: A simple maze (inset) and the average learning curves for Dyna-Q agents varying
in their number of planning steps (n) per real step. The task is to travel from S to G as quickly
as possible.

Figure 8.3 shows why the planning agents found the solution so much faster than
the nonplanning agent. Shown are the policies found by the n = 0 and n = 50 agents
halfway through the second episode. Without planning (n = 0), each episode adds only
one additional step to the policy, and so only one step (the last) has been learned so far.
With planning, again only one step is learned during the first episode, but here during
the second episode an extensive policy has been developed that by the end of the episodewill reach almost back to the start state. This policy is built by the planning process
while the agent is still wandering near the start state. By the end of the third episode acomplete optimal policy will have been found and perfect performance attained.
WITHOUT PLANNING ( n =0)
G

WITH PLANNING ( n =50)
GS

S

Figure 8.3: Policies found by planning and nonplanning Dyna-Q agents halfway through thesecond episode. The arrows indicate the greedy action in each state; if no arrow is shown for astate, then all of its action values were equal. The black square indicates the location of theagent.

166

Chapter 8: Planning and Learning with Tabular Methods

In Dyna-Q, learning and planning are accomplished by exactly the same algorithm,
operating on real experience for learning and on simulated experience for planning.
Because planning proceeds incrementally, it is trivial to intermix planning and acting.
Both proceed as fast as they can. The agent is always reactive and always deliberative,
responding instantly to the latest sensory information and yet always planning in the
background. Also ongoing in the background is the model-learning process. As new
information is gained, the model is updated to better match reality. As the model changes,
the ongoing planning process will gradually compute a di↵erent way of behaving to matchthe new model.
Exercise 8.1 The nonplanning method looks particularly poor in Figure 8.3 because it isa one-step method; a method using multi-step bootstrapping would do better. Do you
think one of the multi-step bootstrapping methods from Chapter 7 could do as well as
the Dyna method? Explain why or why not.
⇤

8.3

When the Model Is Wrong

In the maze example presented in the previous section, the changes in the model wererelatively modest. The model started out empty, and was then filled only with exactly
correct information. In general, we cannot expect to be so fortunate. Models may beincorrect because the environment is stochastic and only a limited number of samples
have been observed, or because the model was learned using function approximation thathas generalized imperfectly, or simply because the environment has changed and its new
behavior has not yet been observed. When the model is incorrect, the planning process islikely to compute a suboptimal policy.
In some cases, the suboptimal policy computed by planning quickly leads to thediscovery and correction of the modeling error. This tends to happen when the modelis optimistic in the sense of predicting greater reward or better state transitions thanare actually possible. The planned policy attempts to exploit these opportunities and indoing so discovers that they do not exist.
Example 8.2: Blocking Maze A maze example illustrating this relatively minor
kind of modeling error and recovery from it is shown in Figure 8.4. Initially, there is a
short path from start to goal, to the right of the barrier, as shown in the upper left of the
figure. After 1000 time steps, the short path is “blocked,” and a longer path is opened up
along the left-hand side of the barrier, as shown in upper right of the figure. The graphshows average cumulative reward for a Dyna-Q agent and an enhanced Dyna-Q+ agent
to be described shortly. The first part of the graph shows that both Dyna agents foundthe short path within 1000 steps. When the environment changed, the graphs become
flat, indicating a period during which the agents obtained no reward because they were
wandering around behind the barrier. After a while, however, they were able to find thenew opening and the new optimal behavior.
Greater difficulties arise when the environment changes to become better than it was
before, and yet the formerly correct policy does not reveal the improvement. In thesecases the modeling error may not be detected for a long time, if ever.

8.3. When the Model Is Wrong

167
G

G

S

S

150

Dyna-Q+

Cumulative
rewardDyna-Q

0
0

1000

2000

3000

Time steps
Figure 8.4: Average performance of Dyna agents on a blocking task. The left environment
was used for the first 1000 steps, the right environment for the rest. Dyna-Q+ is Dyna-Q withan exploration bonus that encourages exploration.

Example 8.3: Shortcut Maze
The problem caused by this kind ofenvironmental change is illustrated
by the maze example shown in Figure 8.5. Initially, the optimal path isto go around the left side of the barrier (upper left). After 3000 steps,
however, a shorter path is opened up
along the right side, without disturbing the longer path (upper right).
The graph shows that the regularDyna-Q agent never switched to theshortcut. In fact, it never realizedthat it existed. Its model said that
there was no shortcut, so the more itplanned, the less likely it was to stepto the right and discover it. Even
with an "-greedy policy, it is very
unlikely that an agent will take somany exploratory actions as to discover the shortcut.

G

G

S

S

400

Dyna-Q+
Dyna-Q

Cumulative
reward

0
0

3000

6000

Time steps
Figure 8.5: Average performance of Dyna agents ona shortcut task. The left environment was used for the
first 3000 steps, the right environment for the rest.

The general problem here is another version of the conflict between exploration andexploitation. In a planning context, exploration means trying actions that improve themodel, whereas exploitation means behaving in the optimal way given the current model.

168

Chapter 8: Planning and Learning with Tabular Methods

We want the agent to explore to find changes in the environment, but not so much that
performance is greatly degraded. As in the earlier exploration/exploitation conflict, there
probably is no solution that is both perfect and practical, but simple heuristics are oftene↵ective.
The Dyna-Q+ agent that did solve the shortcut maze uses one such heuristic. This
agent keeps track for each state–action pair of how many time steps have elapsed since
the pair was last tried in a real interaction with the environment. The more time thathas elapsed, the greater (we might presume) the chance that the dynamics of this pairhas changed and that the model of it is incorrect. To encourage behavior that tests
long-untried actions, a special “bonus reward” is given on simulated experiences involving
these actions. In particular, if the modeled reward for a transition is r, and the transition
has not been tried in ⌧ time p
steps, then planning updates are done as if that transition
produced a reward of r +  ⌧ , for some small . This encourages the agent to keep
testing all accessible state transitions and even to find long sequences of actions in orderto carry out such tests.1 Of course all this testing has its cost, but in many cases, as in the
shortcut maze, this kind of computational curiosity is well worth the extra exploration.
Exercise 8.2 Why did the Dyna agent with exploration bonus, Dyna-Q+, performbetter in the first phase as well as in the second phase of the blocking and shortcut
experiments?
⇤
Exercise 8.3 Careful inspection of Figure 8.5 reveals that the di↵erence between Dyna-Q+
and Dyna-Q narrowed slightly over the first part of the experiment. What is the reason
for this?
⇤
Exercise 8.4 (programming) The exploration bonus described above actually changespthe estimated values of states and actions. Is this necessary? Suppose the bonus  ⌧was used not in updates, but solely in action selection.
That is, suppose the actionpselected was always that for which Q(St , a) +  ⌧ (St , a) was maximal. Carry out agridworld experiment that tests and illustrates the strengths and weaknesses of thisalternate approach.
⇤
Exercise 8.5 How might the tabular Dyna-Q algorithm shown on page 164 be modified
to handle stochastic environments? How might this modification perform poorly onchanging environments such as considered in this section? How could the algorithm bemodified to handle stochastic environments and changing environments?
⇤

8.4

Prioritized Sweeping

In the Dyna agents presented in the preceding sections, simulated transitions are started instate–action pairs selected uniformly at random from all previously experienced pairs. But
a uniform selection is usually not the best; planning can be much more efficient if simulatedtransitions and updates are focused on particular state–action pairs. For example, consider
1 The Dyna-Q+ agent was changed in two other ways as well. First, actions that had never been
tried before from a state were allowed to be considered in the planning step (f) of the Tabular Dyna-Q
algorithm in the box above. Second, the initial model for such actions was that they would lead back tothe same state with a reward of zero.

8.4. Prioritized Sweeping

169

what happens during the second episode of the first maze task (Figure 8.3). At the
beginning of the second episode, only the state–action pair leading directly into the goal
has a positive value; the values of all other pairs are still zero. This means that it ispointless to perform updates along almost all transitions, because they take the agent
from one zero-valued state to another, and thus the updates would have no e↵ect. Onlyan update along a transition into the state just prior to the goal, or from it, will change
any values. If simulated transitions are generated uniformly, then many wasteful updates
will be made before stumbling onto one of these useful ones. As planning progresses, theregion of useful updates grows, but planning is still far less efficient than it would be if
focused where it would do the most good. In the much larger problems that are our real
objective, the number of states is so large that an unfocused search would be extremely
inefficient.
This example suggests that search might be usefully focused by working backward fromgoal states. Of course, we do not really want to use any methods specific to the idea of“goal state.” We want methods that work for general reward functions. Goal states are
just a special case, convenient for stimulating intuition. In general, we want to work backnot just from goal states but from any state whose value has changed. Suppose that thevalues are initially correct given the model, as they were in the maze example prior to
discovering the goal. Suppose now that the agent discovers a change in the environmentand changes its estimated value of one state, either up or down. Typically, this will imply
that the values of many other states should also be changed, but the only useful one-step
updates are those of actions that lead directly into the one state whose value has beenchanged. If the values of these actions are updated, then the values of the predecessor
states may change in turn. If so, then actions leading into them need to be updated, andthen their predecessor states may have changed. In this way one can work backwardfrom arbitrary states that have changed in value, either performing useful updates or
terminating the propagation. This general idea might be termed backward focusing ofplanning computations.
As the frontier of useful updates propagates backward, it often grows rapidly, producing
many state–action pairs that could usefully be updated. But not all of these will beequally useful. The values of some states may have changed a lot, whereas others may
have changed little. The predecessor pairs of those that have changed a lot are more
likely to also change a lot. In a stochastic environment, variations in estimated transition
probabilities also contribute to variations in the sizes of changes and in the urgency withwhich pairs need to be updated. It is natural to prioritize the updates according to ameasure of their urgency, and perform them in order of priority. This is the idea behindprioritized sweeping. A queue is maintained of every state–action pair whose estimatedvalue would change nontrivially if updated , prioritized by the size of the change. Whenthe top pair in the queue is updated, the e↵ect on each of its predecessor pairs is computed.
If the e↵ect is greater than some small threshold, then the pair is inserted in the queuewith the new priority (if there is a previous entry of the pair in the queue, then insertion
results in only the higher priority entry remaining in the queue). In this way the e↵ectsof changes are efficiently propagated backward until quiescence. The full algorithm forthe case of deterministic environments is given in the box on the next page.

170

Chapter 8: Planning and Learning with Tabular Methods

Prioritized sweeping for a deterministic environment
Initialize Q(s, a), M odel(s, a), for all s, a, and P Queue to empty
Loop forever:(a) S
current (nonterminal) state(b) Apolicy(S, Q)(c) Take action A; observe resultant reward, R, and state, S 0(d) M odel(S, A)
R, S 0(e) P|R + maxa Q(S 0 , a) Q(S, A)|.
(f) if P > ✓, then insert S, A into P Queue with priority P(g) Loop repeat n times, while P Queue is not empty:
S, A
f irst(P Queue)
R, S 0M odel(S, A) ⇥⇤Q(S, A)
Q(S, A) + ↵ R + maxa Q(S 0 , a) Q(S, A)
Loop for all S̄, Ā predicted to lead to S:R̄
predicted reward for S̄, Ā, S
P|R̄ + maxa Q(S, a) Q(S̄, Ā)|.
if P > ✓ then insert S̄, Ā into P Queue with priority P

Example 8.4: Prioritized Sweepingon Mazes Prioritized sweeping has beenfound to dramatically increase the speed
at which optimal solutions are found inmaze tasks, often by a factor of 5 to 10.
A typical example is shown to the right. Backups
Updates
Updates
until
until
These data are for a sequence of maze
optimaltasks of exactly the same structure as the optimal
solution
one shown in Figure 8.2, except that they solution
vary in the grid resolution. Prioritizedsweeping maintained a decisive advantage
over unprioritized Dyna-Q. Both systems
made at most n = 5 updates per environmental interaction. Adapted from Pengand Williams (1993).

107

Dyna-Q

106
105

Prioritizedsweeping

104
103
102
10
0

47

94

186

376 752 1504 3008 6016

Gridworld size (#states)

Extensions of prioritized sweeping to stochastic environments are straightforward. The
model is maintained by keeping counts of the number of times each state–action pair has
been experienced and of what the next states were. It is natural then to update each pair
not with a sample update, as we have been using so far, but with an expected update,taking into account all possible next states and their probabilities of occurring.
Prioritized sweeping is just one way of distributing computations to improve planning
efficiency, and probably not the best way. One of prioritized sweeping’s limitations is thatit uses expected updates, which in stochastic environments may waste lots of computation
on low-probability transitions. As we show in the following section, sample updates

8.4. Prioritized Sweeping

Example 8.5

171

Prioritized Sweeping for Rod Maneuvering

The objective in this task is tomaneuver a rod around some awkwardly placed obstacles within alimited rectangular work space to agoal position in the fewest numberof steps. The rod can be translated
along its long axis or perpendicular to that axis, or it can be roGoal
tated in either direction around its
center. The distance of each movement is approximately 1/20 of thework space, and the rotation increment is 10 degrees. Translations
Start
are deterministic and quantized to
one of 20 ⇥ 20 positions. To theright is shown the obstacles and theshortest solution from start to goal,
found by prioritized sweeping. This problem is deterministic, but has four actions
and 14,400 potential states (some of these are unreachable because of the obstacles).
This problem is probably too large to be solved with unprioritized methods. Figure
reprinted from Moore and Atkeson (1993).

can in many cases get closer to the true value function with less computation despite
the variance introduced by sampling. Sample updates can win because they break theoverall backing-up computation into smaller pieces—those corresponding to individualtransitions—which then enables it to be focused more narrowly on the pieces that will
have the largest impact. This idea was taken to what may be its logical limit in the “small
backups” introduced by van Seijen and Sutton (2013). These are updates along a single
transition, like a sample update, but based on the probability of the transition withoutsampling, as in an expected update. By selecting the order in which small updates
are done it is possible to greatly improve planning efficiency beyond that possible withprioritized sweeping.
We have suggested in this chapter that all kinds of state-space planning can be viewed
as sequences of value updates, varying only in the type of update, expected or sample,
large or small, and in the order in which the updates are done. In this section we haveemphasized backward focusing, but this is just one strategy. For example, another would
be to focus on states according to how easily they can be reached from the states thatare visited frequently under the current policy, which might be called forward focusing.
Peng and Williams (1993) and Barto, Bradtke and Singh (1995) have explored versionsof forward focusing, and the methods introduced in the next few sections take it to an
extreme form.

172

8.5

Chapter 8: Planning and Learning with Tabular Methods

Expected vs. Sample Updates

The examples in the previous sections give some idea of the range of possibilities for
combining methods of learning and planning. In the rest of this chapter, we analyze someof the component ideas involved, starting with the relative advantages of expected andsample updates.
Much of this book has been about di↵erent kinds of value-function updates, and wehave considered a great many varieties. Focusing for the moment on one-step updates,
they vary primarily along three binary dimensions. The first two dimensions are whether
they update state values or action values and whether they estimate the value for the
optimal policy or for an arbitrary given policy. These two dimensions give rise to fourclasses of updates for approximating the four value functions, q⇤ , v⇤ , q⇡ , and v⇡ . The
other binary dimension is whether the
updates are expected updates, considerValue
Expected updates
Sample updates(DP)(one-step TD)
ing all possible events that might hap- estimatedpen, or sample updates, considering asssingle sample of what might happen.
⇡
These three binary dimensions give risea⇡Av⇡ (s)
to eight cases, seven of which correR
p rspond to specific algorithms, as showns0
S0
in the figure to the right. (The eighth
policy evaluation
TD(0)case does not seem to correspond to
any useful update.) Any of these ones
step updates can be used in planningmax
methods. The Dyna-Q agents discussedaearlier use q⇤ sample updates, but theyv⇤ (s)
p r
could just as well use q⇤ expected ups0
dates, or either expected or sample q⇡
value iteration
updates. The Dyna-AC system uses v⇡
sample updates together with a learning
s, a
s, a
policy structure (as in Chapter 13). For
p r
Rstochastic problems, prioritized sweeps0
S0
q
(s,
a)
ing is always done using one of the ex⇡⇡pected updates.
a0
A0
When we introduced one-step samq-policy evaluationSarsa
ple updates in Chapter 6, we presented
them as substitutes for expected ups, as, adates. In the absence of a distribution
p rRmodel, expected updates are not poss0
S0
q
(s,
a)
⇤
sible, but sample updates can be donemax
maxusing sample transitions from the envia0
a0
ronment or a sample model. Implicit in
q-value iteration
Q-learningthat point of view is that expected updates, if possible, are preferable to sam- Figure 8.6: Backup diagrams for all the one-stepple updates. But are they? Expected updates considered in this book.

8.5. Expected vs. Sample Updates

173

updates certainly yield a better estimate because they are uncorrupted by sampling error,
but they also require more computation, and computation is often the limiting resourcein planning. To properly assess the relative merits of expected and sample updates forplanning we must control for their di↵erent computational requirements.
For concreteness, consider the expected and sample updates for approximating q⇤ ,and the special case of discrete states and actions, a table-lookup representation of
the approximate value function, Q, and a model in the form of estimated dynamics,p̂(s0 , r |s, a). The expected update for a state–action pair, s, a, is:
h
i
X
0 0
Q(s, a)p̂(s0 , r |s, a) r + max
Q(s
,
a
)
.
(8.1)
0
a

s0 ,r

The corresponding sample update for s, a, given a sample next state and reward, S 0 and
R (from the model), is the Q-learning-like update:hi0 0Q(s, a)Q(s, a) + ↵ R + max
Q(S
,
a
)
Q(s,
a)
,
(8.2)
0
a

where ↵ is the usual positive step-size parameter.
The di↵erence between these expected and sample updates is significant to the extent
that the environment is stochastic, specifically, to the extent that, given a state andaction, many possible next states may occur with various probabilities. If only one next
state is possible, then the expected and sample updates given above are identical (taking↵ = 1). If there are many possible next states, then there may be significant di↵erences.
In favor of the expected update is that it is an exact computation, resulting in a new
Q(s, a) whose correctness is limited only by the correctness of the Q(s0 , a0 ) at successor
states. The sample update is in addition a↵ected by sampling error. On the other hand,the sample update is cheaper computationally because it considers only one next state,
not all possible next states. In practice, the computation required by update operations
is usually dominated by the number of state–action pairs at which Q is evaluated. For aparticular starting pair, s, a, let b be the branching factor (i.e., the number of possible
next states, s0 , for which p̂(s0 |s, a) > 0). Then an expected update of this pair requires
roughly b times as much computation as a sample update.
If there is enough time to complete an expected update, then the resulting estimate is
generally better than that of b sample updates because of the absence of sampling error.
But if there is insufficient time to complete an expected update, then sample updates arealways preferable because they at least make some improvement in the value estimate
with fewer than b updates. In a large problem with many state–action pairs, we are often
in the latter situation. With so many state–action pairs, expected updates of all of themwould take a very long time. Before that we may be much better o↵ with a few sample
updates at many state–action pairs than with expected updates at a few pairs. Given aunit of computational e↵ort, is it better devoted to a few expected updates or to b times
as many sample updates?
Figure 8.7 shows the results of an analysis that suggests an answer to this question. Itshows the estimation error as a function of computation time for expected and sample
updates for a variety of branching factors, b. The case considered is that in which all

174

Chapter 8: Planning and Learning with Tabular Methods

1
expected
updatessampleupdatesb = 2 (branching factor)

RMS error
in valueestimateb =10
b =100b =1000
b =10,000

0
0

1b
0

2b
0

Q(s , a ) computations
Number of max
0a

Figure 8.7: Comparison of efficiency of expected and sample updates.

b successor states are equally likely and in which the error in the initial estimate is
1. The values at the next states are assumed correct, so the expected update reduces
the error to zeroq upon its completion. In this case, sample updates reduce the error

according to bbt1 where t is the number of sample updates that have been performed(assuming sample averages, i.e., ↵ = 1/t). The key observation is that for moderately
large b the error falls dramatically with a tiny fraction of b updates. For these cases,
many state–action pairs could have their values improved dramatically, to within a few
percent of the e↵ect of an expected update, in the same time that a single state–action
pair could undergo an expected update.
The advantage of sample updates shown in Figure 8.7 is probably an underestimate of
the real e↵ect. In a real problem, the values of the successor states would be estimatesthat are themselves updated. By causing estimates to be more accurate sooner, sample
updates will have a second advantage in that the values backed up from the successor
states will be more accurate. These results suggest that sample updates are likely to besuperior to expected updates on problems with large stochastic branching factors andtoo many states to be solved exactly.
Exercise 8.6 The analysis above assumed that all of the b possible next states wereequally likely to occur. Suppose instead that the distribution was highly skewed, thatsome of the b states were much more likely to occur than most. Would this strengthen or
weaken the case for sample updates over expected updates? Support your answer.
⇤

8.6

Trajectory Sampling

In this section we compare two ways of distributing updates. The classical approach, fromdynamic programming, is to perform sweeps through the entire state (or state–action)
space, updating each state (or state–action pair) once per sweep. This is problematic

8.6. Trajectory Sampling175

on large tasks because there may not be time to complete even one sweep. In manytasks the vast majority of the states are irrelevant because they are visited only undervery poor policies or with very low probability. Exhaustive sweeps implicitly devote
equal time to all parts of the state space rather than focusing where it is needed. As wediscussed in Chapter 4, exhaustive sweeps and the equal treatment of all states that they
imply are not necessary properties of dynamic programming. In principle, updates canbe distributed any way one likes (to assure convergence, all states or state–action pairs
must be visited in the limit an infinite number of times; although an exception to this isdiscussed in Section 8.7 below), but in practice exhaustive sweeps are often used.
The second approach is to sample from the state or state–action space according
to some distribution. One could sample uniformly, as in the Dyna-Q agent, but thiswould su↵er from some of the same problems as exhaustive sweeps. More appealing
is to distribute updates according to the on-policy distribution, that is, according to
the distribution observed when following the current policy. One advantage of this
distribution is that it is easily generated; one simply interacts with the model, following
the current policy. In an episodic task, one starts in a start state (or according to thestarting-state distribution) and simulates until the terminal state. In a continuing task,
one starts anywhere and just keeps simulating. In either case, sample state transitions
and rewards are given by the model, and sample actions are given by the current policy.
In other words, one simulates explicit individual trajectories and performs updates at thestate or state–action pairs encountered along the way. We call this way of generating
experience and updates trajectory sampling.
It is hard to imagine any efficient way of distributing updates according to the on-policy
distribution other than by trajectory sampling. If one had an explicit representationof the on-policy distribution, then one could sweep through all states, weighting the
update of each according to the on-policy distribution, but this leaves us again with all
the computational costs of exhaustive sweeps. Possibly one could sample and update
individual state–action pairs from the distribution, but even if this could be done efficiently,
what benefit would this provide over simulating trajectories? Even knowing the on-policy
distribution in an explicit form is unlikely. The distribution changes whenever the policychanges, and computing the distribution requires computation comparable to a complete
policy evaluation. Consideration of such other possibilities makes trajectory samplingseem both efficient and elegant.
Is the on-policy distribution of updates a good one? Intuitively it seems like a good
choice, at least better than the uniform distribution. For example, if you are learning toplay chess, you study positions that might arise in real games, not random positions ofchess pieces. The latter may be valid states, but to be able to accurately value them is adi↵erent skill from evaluating positions in real games. We will also see in Part II that the
on-policy distribution has significant advantages when function approximation is used.
Whether or not function approximation is used, one might expect on-policy focusing tosignificantly improve the speed of planning.
Focusing on the on-policy distribution could be beneficial because it causes vast,
uninteresting parts of the space to be ignored, or it could be detrimental because it causes
the same old parts of the space to be updated over and over. We conducted a small

176

Chapter 8: Planning and Learning with Tabular Methods

experiment to assess the e↵ect empirically. To isolate the e↵ect of the update distribution,
we used entirely one-step expected tabular updates, as defined by (8.1). In the uniform
case, we cycled through all state–action pairs, updating each in place, and in the on-policy
case we simulated episodes, all starting in the same state, updating each state–action pairthat occurred under the current "-greedy policy (" = 0.1). The tasks were undiscounted
episodic tasks, generated randomly as follows. From each of the |S| states, two actions
were possible, each of which resulted in one of b next states, all equally likely, with adi↵erent random selection of b states for each state–action pair. The branching factor, b,
was the same for all state–action pairs. In addition, on all transitions there was a 0.1
probability of transition to the terminal state, ending the episode. The expected reward
on each transition was selected from a Gaussian distribution with mean 0 and variance 1.
At any point in the planning process
one can stop and exhaustively compute
on-policyv⇡˜ (s0 ), the true value of the start stateb=1
3
under the greedy policy, ⇡˜ , given the cur1,000 STATES
rent action-value function Q, as an indiuniform
cation of how well the agent would do on Value of 2
uniforma new episode on which it acted greed- start state
under
b=3
on-policy
ily (all the while assuming the model isgreedypolicycorrect).
1
uniform
b=10
The upper part of the figure to
on-policy
the right shows results averaged over0
200 sample tasks with 1000 states and0
10,00015,00020,0005,000
branching factors of 1, 3, and 10. The
Computation time, in fullbackupsexpected
updatesquality of the policies found is plotted asa function of the number of expected upon-policy
dates completed. In all cases, samplingb=1
according to the on-policy distribution
3
resulted in faster planning initially anduniform
retarded planning in the long run. The
10,000 STATES
Valueofe↵ect was stronger, and the initial pestate 2
riod of faster planning was longer, at startunder
smaller branching factors. In other ex- greedy
policyperiments, we found that these e↵ects
1
also became stronger as the number ofstates increased. For example, the lower
0part of the figure shows results for a0
50,000100,000
150,000
200,000
branching factor of 1 for tasks withexpected
updates
Computation time, in fullbackups
10,000 states. In this case the advantage of on-policy focusing is large and Figure 8.8: Relative efficiency of updates dislong-lasting.
tributed uniformly across the state space versus
All of these results make sense. In the focused on simulated on-policy trajectories, each
short term, sampling according to the starting in the same state. Results are for randomly
on-policy distribution helps by focusing generated tasks of two sizes and various branching
on states that are near descendants of factors, b.

8.7. Real-time Dynamic Programming177

the start state. If there are many states and a small branching factor, this e↵ect will belarge and long-lasting. In the long run, focusing on the on-policy distribution may hurt
because the commonly occurring states all already have their correct values. Samplingthem is useless, whereas sampling other states may actually perform some useful work.
This presumably is why the exhaustive, unfocused approach does better in the long run,at least for small problems. These results are not conclusive because they are only for
problems generated in a particular, random way, but they do suggest that samplingaccording to the on-policy distribution can be a great advantage for large problems, inparticular for problems in which a small subset of the state–action space is visited under
the on-policy distribution.
Exercise 8.7 Some of the graphs in Figure 8.8 seem to be scalloped in their early portions,particularly the upper graph for b = 1 and the uniform distribution. Why do you think
this is? What aspects of the data shown support your hypothesis?
⇤
Exercise 8.8 (programming) Replicate the experiment whose results are shown in thelower part of Figure 8.8, then try the same experiment but with b = 3. Discuss themeaning of your results.
⇤

8.7

Real-time Dynamic Programming

Real-time dynamic programming, or RTDP, is an on-policy trajectory-sampling version of
the value-iteration algorithm of dynamic programming (DP). Because it is closely related
to conventional sweep-based policy iteration, RTDP illustrates in a particularly clear way
some of the advantages that on-policy trajectory sampling can provide. RTDP updates
the values of states visited in actual or simulated trajectories by means of expectedtabular value-iteration updates as defined by (4.10). It is basically the algorithm that
produced the on-policy results shown in Figure 8.8.
The close connection between RTDP and conventional DP makes it possible to derivesome theoretical results by adapting existing theory. RTDP is an example of an asynchronous DP algorithm as described in Section 4.5. Asynchronous DP algorithms are
not organized in terms of systematic sweeps of the state set; they update state values inany order whatsoever, using whatever values of other states happen to be available. In
RTDP, the update order is dictated by the order states are visited in real or simulated
trajectories.
If trajectories can start only from a designatedIrrelevant States:
unreachable from any start state
set of start states, and if you are interested inunder any optimal policy
the prediction problem for a given policy, then on- Start States
policy trajectory sampling allows the algorithm tocompletely skip states that cannot be reached by
the given policy from any of the start states: suchstates are irrelevant to the prediction problem.
For a control problem, where the goal is to find
Relevant States
an optimal policy instead of evaluating a given
reachable from some start state
under some optimal policy
policy, there might well be states that cannot be

178

Chapter 8: Planning and Learning with Tabular Methods

reached by any optimal policy from any of the start states, and there is no need to specify
optimal actions for these irrelevant states. What is needed is an optimal partial policy,
meaning a policy that is optimal for the relevant states but can specify arbitrary actions,or even be undefined, for the irrelevant states.
But finding such an optimal partial policy with an on-policy trajectory-sampling
control method, such as Sarsa (Section 6.4), in general requires visiting all state–action
pairs—even those that will turn out to be irrelevant—an infinite number of times. Thiscan be done, for example, by using exploring starts (Section 5.3). This is true for RTDP
as well: for episodic tasks with exploring starts, RTDP is an asynchronous value-iteration
algorithm that converges to optimal policies for discounted finite MDPs (and for theundiscounted case under certain conditions). Unlike the situation for a prediction problem,
it is generally not possible to stop updating any state or state–action pair if convergenceto an optimal policy is important.
The most interesting result for RTDP is that for certain types of problems satisfying
reasonable conditions, RTDP is guaranteed to find a policy that is optimal on the relevant
states without visiting every state infinitely often, or even without visiting some states atall. Indeed, in some problems, only a small fraction of the states need to be visited. Thiscan be a great advantage for problems with very large state sets, where even a single
sweep may not be feasible.
The tasks for which this result holds are undiscounted episodic tasks for MDPs withabsorbing goal states that generate zero rewards, as described in Section 3.4. At every stepof a real or simulated trajectory, RTDP selects a greedy action (breaking ties randomly)
and applies the expected value-iteration update operation to the current state. It canalso update the values of an arbitrary collection of other states at each step; for example,
it can update the values of states visited in a limited-horizon look-ahead search from thecurrent state.
For these problems, with each episode beginning in a state randomly chosen from theset of start states and ending at a goal state, RTDP converges with probability one to apolicy that is optimal for all the relevant states provided: (1) the initial value of every
goal state is zero, (2) there exists at least one policy that guarantees that a goal state
will be reached with probability one from any start state, (3) all rewards for transitions
from non-goal states are strictly negative, and (4) all the initial values are equal to, or
greater than, their optimal values (which can be satisfied by simply setting the initialvalues of all states to zero). This result was proved by Barto, Bradtke, and Singh (1995)
by combining results for asynchronous DP with results about a heuristic search algorithm
known as learning real-time A* due to Korf (1990).
Tasks having these properties are examples of stochastic optimal path problems, which
are usually stated in terms of cost minimization instead of as reward maximization as
we do here. Maximizing the negative returns in our version is equivalent to minimizing
the costs of paths from a start state to a goal state. Examples of this kind of task areminimum-time control tasks, where each time step required to reach a goal produces areward of 1, or problems like the Golf example in Section 3.5, whose objective is to hit
the hole with the fewest strokes.

8.7. Real-time Dynamic Programming

179

Example 8.6: RTDP on the Racetrack The racetrack problem of Exercise 5.12
(page 111) is a stochastic optimal path problem. Comparing RTDP and the conventional
DP value iteration algorithm on an example racetrack problem illustrates some of theadvantages of on-policy trajectory sampling.
Recall from the exercise that an agent has to learn how to drive a car around a turnlike those shown in Figure 5.5 and cross the finish line as quickly as possible while staying
on the track. Start states are all the zero-speed states on the starting line; the goal states
are all the states that can be reached in one time step by crossing the finish line frominside the track. Unlike Exercise 5.12, here there is no limit on the car’s speed, so thestate set is potentially infinite. However, the set of states that can be reached from theset of start states via any policy is finite and can be considered to be the state set of the
problem. Each episode begins in a randomly selected start state and ends when the carcrosses the finish line. The rewards are 1 for each step until the car crosses the finishline. If the car hits the track boundary, it is moved back to a random start state, and theepisode continues.
A racetrack similar to the small racetrack on the left of Figure 5.5 has 9,115 states
reachable from start states by any policy, only 599 of which are relevant, meaning thatthey are reachable from some start state via some optimal policy. (The number of relevant
states was estimated by counting the states visited while executing optimal actions for
107 episodes.)The table below compares solving this task by conventional DP and by RTDP. These
results are averages over 25 runs, each begun with a di↵erent random number seed.
Conventional DP in this case is value iteration using exhaustive sweeps of the state set,with values updated one state at a time in place, meaning that the update for each state
uses the most recent values of the other states (This is the Gauss-Seidel version of value
iteration, which was found to be approximately twice as fast as the Jacobi version on
this problem. See Section 4.8.) No special attention was paid to the ordering of the
updates; other orderings could have produced faster convergence. Initial values were all
zero for each run of both methods. DP was judged to have converged when the maximum
change in a state value over a sweep was less than 10 4 , and RTDP was judged to have
converged when the average time to cross the finish line over 20 episodes appeared tostabilize at an asymptotic number of steps. This version of RTDP updated only the valueof the current state on each step.

Average computation to convergence
Average number of updates to convergence
Average number of updates per episode% of states updated  100 times% of states updated  10 times% of states updated 0 timesDP
28 sweeps
252,784
—
—
—
—

RTDP
4000 episodes
127,600
31.9
98.45
80.51
3.18

Both methods produced policies averaging between 14 and 15 steps to cross the finish
line, but RTDP required only roughly half of the updates that DP did. This is the resultof RTDP’s on-policy trajectory sampling. Whereas the value of every state was updated

180

Chapter 8: Planning and Learning with Tabular Methods

in each sweep of DP, RTDP focused updates on fewer states. In an average run, RTDP
updated the values of 98.45% of the states no more than 100 times and 80.51% of thestates no more than 10 times; the values of about 290 states were not updated at all inan average run.
Another advantage of RTDP is that as the value function approaches the optimalvalue function v⇤ , the policy used by the agent to generate trajectories approaches anoptimal policy because it is always greedy with respect to the current value function.
This is in contrast to the situation in conventional value iteration. In practice, value
iteration terminates when the value function changes by only a small amount in a sweep,
which is how we terminated it to obtain the results in the table above. At this point,
the value function closely approximates v⇤ , and a greedy policy is close to an optimal
policy. However, it is possible that policies that are greedy with respect to the latest
value function were optimal, or nearly so, long before value iteration terminates. (Recall
from Chapter 4 that optimal policies can be greedy with respect to many di↵erent
value functions, not just v⇤ .) Checking for the emergence of an optimal policy before
value iteration converges is not a part of the conventional DP algorithm and requiresconsiderable additional computation.
In the racetrack example, by running many test episodes after each DP sweep, withactions selected greedily according to the result of that sweep, it was possible to estimate
the earliest point in the DP computation at which the approximated optimal evaluation
function was good enough so that the corresponding greedy policy was nearly optimal.
For this racetrack, a close-to-optimal policy emerged after 15 sweeps of value iteration, or
after 136,725 value-iteration updates. This is considerably less than the 252,784 updates
DP needed to converge to v⇤ , but still more than the 127,600 updates RTDP required.
Although these simulations are certainly not definitive comparisons of the RTDP withconventional sweep-based value iteration, they illustrate some of advantages of on-policy
trajectory sampling. Whereas conventional value iteration continued to update the valueof all the states, RTDP strongly focused on subsets of the states that were relevant to
the problem’s objective. This focus became increasingly narrow as learning continued.
Because the convergence theorem for RTDP applies to the simulations, we know that
RTDP eventually would have focused only on relevant states, i.e., on states making up
optimal paths. RTDP achieved nearly optimal control with about 50% of the computation
required by sweep-based value iteration.

8.8

Planning at Decision Time

Planning can be used in at least two ways. The one we have considered so far in thischapter, typified by dynamic programming and Dyna, is to use planning to gradually
improve a policy or value function on the basis of simulated experience obtained from amodel (either a sample or a distribution model). Selecting actions is then a matter ofcomparing the current state’s action values obtained from a table in the tabular case wehave thus far considered, or by evaluating a mathematical expression in the approximate
methods we consider in Part II below. Well before an action is selected for any current
state St , planning has played a part in improving the table entries, or the function

8.9. Heuristic Search

181

approximation parameters, needed to select actions for many states, including St . Used
this way, planning is not focused on the current state. We call planning used in this way
background planning.
The other way to use planning is to begin and complete it after encountering eachnew state St , as a computation whose output is the selection of a single action At ; on
the next step planning begins anew with St+1 to produce At+1 , and so on. The simplest,
and almost degenerate, example of this use of planning is when only state values areavailable, and an action is selected by comparing the values of model-predicted next states
for each action (or by comparing the values of afterstates as in the tic-tac-toe example
in Chapter 1). More generally, planning used in this way can look much deeper than
one-step-ahead and evaluate action choices leading to many di↵erent predicted state andreward trajectories. Unlike the first use of planning, here planning focuses on a particularstate. We call this decision-time planning.
These two ways of thinking about planning—using simulated experience to gradually
improve a policy or value function, or using simulated experience to select an action forthe current state—can blend together in natural and interesting ways, but they havetended to be studied separately, and that is a good way to first understand them. Let us
now take a closer look at decision-time planning.
Even when planning is only done at decision time, we can still view it, as we didin Section 8.1, as proceeding from simulated experience to updates and values, andultimately to a policy. It is just that now the values and policy are specific to the current
state and the action choices available there, so much so that the values and policy createdby the planning process are typically discarded after being used to select the current
action. In many applications this is not a great loss because there are very many states
and we are unlikely to return to the same state for a long time. In general, one may
want to do a mix of both: focus planning on the current state and store the resultsof planning so as to be that much farther along should one return to the same statelater. Decision-time planning is most useful in applications in which fast responses are
not required. In chess playing programs, for example, one may be permitted seconds or
minutes of computation for each move, and strong programs may plan dozens of moves
ahead within this time. On the other hand, if low latency action selection is the priority,then one is generally better o↵ doing planning in the background to compute a policythat can then be rapidly applied to each newly encountered state.

8.9

Heuristic Search

The classical state-space planning methods in artificial intelligence are decision-timeplanning methods collectively known as heuristic search. In heuristic search, for eachstate encountered, a large tree of possible continuations is considered. The approximate
value function is applied to the leaf nodes and then backed up toward the current stateat the root. The backing up within the search tree is just the same as in the expected
updates with maxes (those for v⇤ and q⇤ ) discussed throughout this book. The backing
up stops at the state–action nodes for the current state. Once the backed-up values ofthese nodes are computed, the best of them is chosen as the current action, and then all
backed-up values are discarded.

182

Chapter 8: Planning and Learning with Tabular Methods

In conventional heuristic search no e↵ort is made to save the backed-up values bychanging the approximate value function. In fact, the value function is generally designed
by people and never changed as a result of search. However, it is natural to consider
allowing the value function to be improved over time, using either the backed-up values
computed during heuristic search or any of the other methods presented throughout
this book. In a sense we have taken this approach all along. Our greedy, "-greedy, andUCB (Section 2.7) action-selection methods are not unlike heuristic search, albeit on asmaller scale. For example, to compute the greedy action given a model and a state-valuefunction, we must look ahead from each possible action to each possible next state, take
into account the rewards and estimated values, and then pick the best action. Just as
in conventional heuristic search, this process computes backed-up values of the possible
actions, but does not attempt to save them. Thus, heuristic search can be viewed as anextension of the idea of a greedy policy beyond a single step.
The point of searching deeper than one step is to obtain better action selections. If one
has a perfect model and an imperfect action-value function, then in fact deeper search
will usually yield better policies.2 Certainly, if the search is all the way to the end of
the episode, then the e↵ect of the imperfect value function is eliminated, and the action
determined in this way must be optimal. If the search is of sufficient depth k such that k
is very small, then the actions will be correspondingly near optimal. On the other hand,the deeper the search, the more computation is required, usually resulting in a slower
response time. A good example is provided by Tesauro’s grandmaster-level backgammon
player, TD-Gammon (Section 16.1). This system used TD learning to learn an afterstatevalue function through many games of self-play, using a form of heuristic search to makeits moves. As a model, TD-Gammon used a priori knowledge of the probabilities of dicerolls and the assumption that the opponent always selected the actions that TD-Gammon
rated as best for it. Tesauro found that the deeper the heuristic search, the better the
moves made by TD-Gammon, but the longer it took to make each move. Backgammon
has a large branching factor, yet moves must be made within a few seconds. It was
only feasible to search ahead selectively a few steps, but even so the search resulted insignificantly better action selections.
We should not overlook the most obvious way in which heuristic search focuses updates:on the current state. Much of the e↵ectiveness of heuristic search is due to its search tree
being tightly focused on the states and actions that might immediately follow the current
state. You may spend more of your life playing chess than checkers, but when you playcheckers, it pays to think about checkers and about your particular checkers position,
your likely next moves, and successor positions. No matter how you select actions, itis these states and actions that are of highest priority for updates and where you mosturgently want your approximate value function to be accurate. Not only should yourcomputation be preferentially devoted to imminent events, but so should your limited
memory resources. In chess, for example, there are far too many possible positions tostore distinct value estimates for each of them, but chess programs based on heuristic
search can easily store distinct estimates for the millions of positions they encounter
2 There are interesting exceptions to this (see Pearl, 1984).

8.10. Rollout Algorithms

183

looking ahead from a single position. This great focusing of memory and computational
resources on the current decision is presumably the reason why heuristic search can be soe↵ective.
The distribution of updates can be altered in similar ways to focus on the current
state and its likely successors. As a limiting case we might use exactly the methods ofheuristic search to construct a search tree, and then perform the individual, one-step
updates from bottom up, as suggested by Figure 8.9. If the updates are ordered in thisway and a tabular representation is used, then exactly the same overall update would
be achieved as in depth-first heuristic search. Any state-space search can be viewed inthis way as the piecing together of a large number of individual one-step updates. Thus,
the performance improvement observed with deeper searches is not due to the use ofmultistep updates as such. Instead, it is due to the focus and concentration of updates
on states and actions immediately downstream from the current state. By devoting alarge amount of computation specifically relevant to the candidate actions, decision-timeplanning can produce better decisions than can be produced by relying on unfocused
updates.

3

1

10

8

6

2

4

5

9

7

Figure 8.9: Heuristic search can be implemented as a sequence of one-step updates (shown
here outlined in blue) backing up values from the leaf nodes toward the root. The ordering
shown here is for a selective depth-first search.

8.10

Rollout Algorithms

Rollout algorithms are decision-time planning algorithms based on Monte Carlo control
applied to simulated trajectories that all begin at the current environment state. They
estimate action values for a given policy by averaging the returns of many simulated
trajectories that start with each possible action and then follow the given policy. Whenthe action-value estimates are considered to be accurate enough, the action (or one of the184

Chapter 8: Planning and Learning with Tabular Methods

actions) having the highest estimated value is executed, after which the process is carriedout anew from the resulting next state. As explained by Tesauro and Galperin (1997),
who experimented with rollout algorithms for playing backgammon, the term “rollout”
comes from estimating the value of a backgammon position by playing out, i.e., “rollingout,” the position many times to the game’s end with randomly generated sequences ofdice rolls, where the moves of both players are made by some fixed policy.
Unlike the Monte Carlo control algorithms described in Chapter 5, the goal of arollout algorithm is not to estimate a complete optimal action-value function, q⇤ , or acomplete action-value function, q⇡ , for a given policy ⇡. Instead, they produce Monte
Carlo estimates of action values only for each current state and for a given policy usuallycalled the rollout policy. As decision-time planning algorithms, rollout algorithms make
immediate use of these action-value estimates, then discard them. This makes rollout
algorithms relatively simple to implement because there is no need to sample outcomesfor every state-action pair, and there is no need to approximate a function over either
the state space or the state-action space.
What then do rollout algorithms accomplish? The policy improvement theorem
described in Section 4.2 tells us that given any two policies ⇡ and ⇡ 0 that are identical
except that ⇡ 0 (s) = a 6= ⇡(s) for some state s, if q⇡ (s, a) v⇡ (s), then policy ⇡ 0 is as goodas, or better, than ⇡. Moreover, if the inequality is strict, then ⇡ 0 is in fact better than ⇡.
This applies to rollout algorithms where s is the current state and ⇡ is the rollout policy.
Averaging the returns of the simulated trajectories produces estimates of q⇡ (s, a0 ) for
each action a0 2 A(s). Then the policy that selects an action in s that maximizes theseestimates and thereafter follows ⇡ is a good candidate for a policy that improves over⇡. The result is like one step of the policy-iteration algorithm of dynamic programming
discussed in Section 4.3 (though it is more like one step of asynchronous value iteration
described in Section 4.5 because it changes the action for just the current state).
In other words, the aim of a rollout algorithm is to improve upon the rollout policy;
not to find an optimal policy. Experience has shown that rollout algorithms can be
surprisingly e↵ective. For example, Tesauro and Galperin (1997) were surprised by thedramatic improvements in backgammon playing ability produced by the rollout method.
In some applications, a rollout algorithm can produce good performance even if therollout policy is completely random. But the performance of the improved policy depends
on properties of the rollout policy and the ranking of actions produced by the Monte
Carlo value estimates. Intuition suggests that the better the rollout policy and the moreaccurate the value estimates, the better the policy produced by a rollout algorithm islikely be (but see Gelly and Silver, 2007).
This involves important tradeo↵s because better rollout policies typically mean that
more time is needed to simulate enough trajectories to obtain good value estimates.
As decision-time planning methods, rollout algorithms usually have to meet strict time
constraints. The computation time needed by a rollout algorithm depends on the numberof actions that have to be evaluated for each decision, the number of time steps in thesimulated trajectories needed to obtain useful sample returns, the time it takes the rollout
policy to make decisions, and the number of simulated trajectories needed to obtain goodMonte Carlo action-value estimates.

8.11. Monte Carlo Tree Search

185

Balancing these factors is important in any application of rollout methods, though there
are several ways to ease the challenge. Because the Monte Carlo trials are independent of
one another, it is possible to run many trials in parallel on separate processors. Anotherapproach is to truncate the simulated trajectories short of complete episodes, correcting
the truncated returns by means of a stored evaluation function (which brings into play
all that we have said about truncated returns and updates in the preceding chapters).
It is also possible, as Tesauro and Galperin (1997) suggest, to monitor the Monte Carlosimulations and prune away candidate actions that are unlikely to turn out to be thebest, or whose values are close enough to that of the current best that choosing theminstead would make no real di↵erence (though Tesauro and Galperin point out that thiswould complicate a parallel implementation).
We do not ordinarily think of rollout algorithms as learning algorithms because they
do not maintain long-term memories of values or policies. However, these algorithms takeadvantage of some of the features of reinforcement learning that we have emphasizedin this book. As instances of Monte Carlo control, they estimate action values by
averaging the returns of a collection of sample trajectories, in this case trajectories ofsimulated interactions with a sample model of the environment. In this way they arelike reinforcement learning algorithms in avoiding the exhaustive sweeps of dynamic
programming by trajectory sampling, and in avoiding the need for distribution models
by relying on sample, instead of expected, updates. Finally, rollout algorithms takeadvantage of the policy improvement property by acting greedily with respect to theestimated action values.

8.11

Monte Carlo Tree SearchMonte Carlo Tree Search (MCTS) is a recent and strikingly successful example of decisiontime planning. At its base, MCTS is a rollout algorithm as described above, but enhancedby the addition of a means for accumulating value estimates obtained from the Monte
Carlo simulations in order to successively direct simulations toward more highly-rewarding
trajectories. MCTS is largely responsible for the improvement in computer Go froma weak amateur level in 2005 to a grandmaster level (6 dan or more) in 2015. Manyvariations of the basic algorithm have been developed, including a variant that we discussin Section 16.6 that was critical for the stunning 2016 victories of the program AlphaGoover an 18-time world champion Go player. MCTS has proved to be e↵ective in a wide
variety of competitive settings, including general game playing (e.g., see Finnsson andBjörnsson, 2008; Genesereth and Thielscher, 2014), but it is not limited to games; it canbe e↵ective for single-agent sequential decision problems if there is an environment model
simple enough for fast multistep simulation.
MCTS is executed after encountering each new state to select the agent’s action forthat state; it is executed again to select the action for the next state, and so on. As in arollout algorithm, each execution is an iterative process that simulates many trajectories
starting from the current state and running to a terminal state (or until discounting
makes any further reward negligible as a contribution to the return). The core ideaof MCTS is to successively focus multiple simulations starting at the current state by

186

Chapter 8: Planning and Learning with Tabular Methods

extending the initial portions of trajectories that have received high evaluations fromearlier simulations. MCTS does not have to retain approximate value functions or policies
from one action selection to the next, though in many implementations it retains selectedaction values likely to be useful for its next execution.
For the most part, the actions in the simulated trajectories are generated using a simple
policy, usually called a rollout policy as it is for simpler rollout algorithms. When both
the rollout policy and the model do not require a lot of computation, many simulated
trajectories can be generated in a short period of time. As in any tabular Monte Carlo
method, the value of a state–action pair is estimated as the average of the (simulated)
returns from that pair. Monte Carlo value estimates are maintained only for the subsetof state–action pairs that are most likely to be reached in a few steps, which form a tree
rooted at the current state, as illustrated in Figure 8.10. MCTS incrementally extends
the tree by adding nodes representing states that look promising based on the resultsof the simulated trajectories. Any simulated trajectory will pass through the tree andthen exit it at some leaf node. Outside the tree and at the leaf nodes the rollout policy isused for action selections, but at the states inside the tree something better is possible.
For these states we have value estimates for at least some of the actions, so we can pickamong them using an informed policy, called the tree policy, that balances exploration
Repeat while time remains

Selection

Expansion

Tree
PolicySimulation

BackupRollout
PolicyFigure 8.10: Monte Carlo Tree Search. When the environment changes to a new state, MCTS
executes as many iterations as possible before an action needs to be selected, incrementallybuilding a tree whose root node represents the current state. Each iteration consists of the four
operations Selection, Expansion (though possibly skipped on some iterations), Simulation,
and Backup, as explained in the text and illustrated by the bold arrows in the trees. Adaptedfrom Chaslot, Bakkes, Szita, and Spronck (2008).

8.11. Monte Carlo Tree Search

187

and exploitation. For example, the tree policy could select actions using an "-greedy or
UCB selection rule (Chapter 2).
In more detail, each iteration of a basic version of MCTS consists of the following four
steps as illustrated in Figure 8.10:
1. Selection. Starting at the root node, a tree policy based on the action values
attached to the edges of the tree traverses the tree to select a leaf node.
2. Expansion. On some iterations (depending on details of the application), the tree
is expanded from the selected leaf node by adding one or more child nodes reachedfrom the selected node via unexplored actions.
3. Simulation. From the selected node, or from one of its newly-added child nodes(if any), simulation of a complete episode is run with actions selected by the rollout
policy. The result is a Monte Carlo trial with actions selected first by the tree
policy and beyond the tree by the rollout policy.
4. Backup. The return generated by the simulated episode is backed up to update,or to initialize, the action values attached to the edges of the tree traversed by
the tree policy in this iteration of MCTS. No values are saved for the states andactions visited by the rollout policy beyond the tree. Figure 8.10 illustrates this byshowing a backup from the terminal state of the simulated trajectory directly to thestate–action node in the tree where the rollout policy began (though in general, theentire return over the simulated trajectory is backed up to this state–action node).
MCTS continues executing these four steps, starting each time at the tree’s root node,
until no more time is left, or some other computational resource is exhausted. Then,
finally, an action from the root node (which still represents the current state of theenvironment) is selected according to some mechanism that depends on the accumulated
statistics in the tree; for example, it may be an action having the largest action valueof all the actions available from the root state, or perhaps the action with the largest
visit count to avoid selecting outliers. This is the action MCTS actually selects. Afterthe environment transitions to a new state, MCTS is run again, sometimes startingwith a tree of a single root node representing the new state, but often starting with atree containing any descendants of this node left over from the tree constructed by theprevious execution of MCTS; all the remaining nodes are discarded, along with the action
values associated with them.
MCTS was first proposed to select moves in programs playing two-person competitive
games, such as Go. For game playing, each simulated episode is one complete play of thegame in which both players select actions by the tree and rollout policies. Section 16.6
describes an extension of MCTS used in the AlphaGo program that combines the Monte
Carlo evaluations of MCTS with action values learned by a deep artificial neural network
via self-play reinforcement learning.
Relating MCTS to the reinforcement learning principles we describe in this bookprovides some insight into how it achieves such impressive results. At its base, MCTS isa decision-time planning algorithm based on Monte Carlo control applied to simulations

188

Chapter 8: Planning and Learning with Tabular Methods

that start from the root state; that is, it is a kind of rollout algorithm as described in
the previous section. It therefore benefits from online, incremental, sample-based value
estimation and policy improvement. Beyond this, it saves action-value estimates attachedto the tree edges and updates them using reinforcement learning’s sample updates. Thishas the e↵ect of focusing the Monte Carlo trials on trajectories whose initial segments
are common to high-return trajectories previously simulated. Further, by incrementally
expanding the tree, MCTS e↵ectively grows a lookup table to store a partial action-valuefunction, with memory allocated to the estimated values of state–action pairs visited in
the initial segments of high-yielding sample trajectories. MCTS thus avoids the problem
of globally approximating an action-value function while it retains the benefit of using
past experience to guide exploration.
The striking success of decision-time planning by MCTS has deeply influenced artificial
intelligence, and many researchers are studying modifications and extensions of the basic
procedure for use in both games and single-agent applications.

8.12

Summary of the Chapter

Planning requires a model of the environment. A distribution model consists of theprobabilities of next states and rewards for possible actions; a sample model produces
single transitions and rewards generated according to these probabilities. Dynamic
programming requires a distribution model because it uses expected updates, which involve
computing expectations over all the possible next states and rewards. A sample model,on the other hand, is what is needed to simulate interacting with the environment during
which sample updates, like those used by many reinforcement learning algorithms, can beused. Sample models are generally much easier to obtain than distribution models.
We have presented a perspective emphasizing the surprisingly close relationships between planning optimal behavior and learning optimal behavior. Both involve estimating
the same value functions, and in both cases it is natural to update the estimates incrementally, in a long series of small backing-up operations. This makes it straightforward
to integrate learning and planning processes simply by allowing both to update the same
estimated value function. In addition, any of the learning methods can be converted intoplanning methods simply by applying them to simulated (model-generated) experiencerather than to real experience. In this case learning and planning become even moresimilar; they are possibly identical algorithms operating on two di↵erent sources ofexperience.
It is straightforward to integrate incremental planning methods with acting and modellearning. Planning, acting, and model-learning interact in a circular fashion (as in
the diagram on page 162), each producing what the other needs to improve; no otherinteraction among them is either required or prohibited. The most natural approach
is for all processes to proceed asynchronously and in parallel. If the processes must
share computational resources, then the division can be handled almost arbitrarily—by
whatever organization is most convenient and efficient for the task at hand.
In this chapter we have touched upon a number of dimensions of variation amongstate-space planning methods. One dimension is the variation in the size of updates. The

8.13. Summary of Part I: Dimensions

189

smaller the updates, the more incremental the planning methods can be. Among thesmallest updates are one-step sample updates, as in Dyna. Another important dimension
is the distribution of updates, that is, of the focus of search. Prioritized sweeping focuses
backward on the predecessors of states whose values have recently changed. On-policy
trajectory sampling focuses on states or state–action pairs that the agent is likely to
encounter when controlling its environment. This can allow computation to skip overparts of the state space that are irrelevant to the prediction or control problem. Realtime dynamic programming, an on-policy trajectory sampling version of value iteration,
illustrates some of the advantages this strategy has over conventional sweep-based policy
iteration.
Planning can also focus forward from pertinent states, such as states actually encountered during an agent-environment interaction. The most important form of this is whenplanning is done at decision time, that is, as part of the action-selection process. Classicalheuristic search as studied in artificial intelligence is an example of this. Other examples
are rollout algorithms and Monte Carlo Tree Search that benefit from online, incremental,
sample-based value estimation and policy improvement.

8.13

Summary of Part I: Dimensions

This chapter concludes Part I of this book. In it we have tried to present reinforcement
learning not as a collection of individual methods, but as a coherent set of ideas cutting
across methods. Each idea can be viewed as a dimension along which methods vary. The
set of such dimensions spans a large space of possible methods. By exploring this spaceat the level of dimensions we hope to obtain the broadest and most lasting understanding.
In this section we use the concept of dimensions in method space to recapitulate the viewof reinforcement learning developed so far in this book.
All of the methods we have explored so far in this book have three key ideas in common:
first, they all seek to estimate value functions; second, they all operate by backing up
values along actual or possible state trajectories; and third, they all follow the general
strategy of generalized policy iteration (GPI), meaning that they maintain an approximate
value function and an approximate policy, and they continually try to improve each on thebasis of the other. These three ideas are central to the subjects covered in this book. We
suggest that value functions, backing up value updates, and GPI are powerful organizing
principles potentially relevant to any model of intelligence, whether artificial or natural.
Two of the most important dimensions along which the methods vary are shown in
Figure 8.11. These dimensions have to do with the kind of update used to improve thevalue function. The horizontal dimension is whether they are sample updates (based on asample trajectory) or expected updates (based on a distribution of possible trajectories).
Expected updates require a distribution model, whereas sample updates need only asample model, or can be done from actual experience with no model at all (anotherdimension of variation). The vertical dimension of Figure 8.11 corresponds to the depthof updates, that is, to the degree of bootstrapping. At three of the four corners of thespace are the three primary methods for estimating values: dynamic programming, TD,
and Monte Carlo. Along the left edge of the space are the sample-update methods,

190

Chapter 8: Planning and Learning with Tabular Methods

Temporaldifference
learning

width
of update

Dynamic
programming

depth(length)of update

Exhaustive
search

Monte
Carlo

...
Figure 8.11: A slice through the space of reinforcement learning methods, highlighting thetwo of the most important dimensions explored in Part I of this book: the depth and width of
the updates.

ranging from one-step TD updates to full-return Monte Carlo updates. Between theseis a spectrum including methods based on n-step updates (and in Chapter 12 we will
extend this to mixtures of n-step updates such as the -updates implemented by eligibility
traces).
Dynamic programming methods are shown in the extreme upper-right corner of thespace because they involve one-step expected updates. The lower-right corner is theextreme case of expected updates so deep that they run all the way to terminal states(or, in a continuing task, until discounting has reduced the contribution of any further
rewards to a negligible level). This is the case of exhaustive search. Intermediate methods
along this dimension include heuristic search and related methods that search and update
up to a limited depth, perhaps selectively. There are also methods that are intermediatealong the horizontal dimension. These include methods that mix expected and sample
updates, as well as the possibility of methods that mix samples and distributions within
a single update. The interior of the square is filled in to represent the space of all suchintermediate methods.
A third dimension that we have emphasized in this book is the binary distinction
between on-policy and o↵-policy methods. In the former case, the agent learns the valuefunction for the policy it is currently following, whereas in the latter case it learns the8.13. Summary of Part I: Dimensions

191

value function for the policy for a di↵erent policy, often the one that the agent currently
thinks is best. The policy generating behavior is typically di↵erent from what is currently
thought best because of the need to explore. This third dimension might be visualized as
perpendicular to the plane of the page in Figure 8.11.
In addition to the three dimensions just discussed, we have identified a number ofothers throughout the book:
Definition of return Is the task episodic or continuing, discounted or undiscounted?
Action values vs. state values vs. afterstate values What kind of values shouldbe estimated? If only state values are estimated, then either a model or a separate
policy (as in actor–critic methods) is required for action selection.
Action selection/exploration How are actions selected to ensure a suitable trade-o↵
between exploration and exploitation? We have considered only the simplest ways todo this: "-greedy, optimistic initialization of values, soft-max, and upper confidencebound.
Synchronous vs. asynchronous Are the updates for all states performed simultaneously or one by one in some order?
Real vs. simulated Should one update based on real experience or simulated experience? If both, how much of each?
Location of updates What states or state–action pairs should be updated? Modelfree methods can choose only among the states and state–action pairs actuallyencountered, but model-based methods can choose arbitrarily. There are many
possibilities here.
Timing of updates Should updates be done as part of selecting actions, or only afterward?
Memory for updates How long should updated values be retained? Should they beretained permanently, or only while computing an action selection, as in heuristic
search?
Of course, these dimensions are neither exhaustive nor mutually exclusive. Individual
algorithms di↵er in many other ways as well, and many algorithms lie in several places
along several dimensions. For example, Dyna methods use both real and simulated
experience to a↵ect the same value function. It is also perfectly sensible to maintainmultiple value functions computed in di↵erent ways or over di↵erent state and action
representations. These dimensions do, however, constitute a coherent set of ideas fordescribing and exploring a wide space of possible methods.
The most important dimension not mentioned here, and not covered in Part I of
this book, is that of function approximation. Function approximation can be viewed as
an orthogonal spectrum of possibilities ranging from tabular methods at one extreme
through state aggregation, a variety of linear methods, and then a diverse set of nonlinear
methods. This dimension is explored in Part II.

192

Chapter 8: Planning and Learning with Tabular Methods

Bibliographical and Historical Remarks
8.1

The overall view of planning and learning presented here has developed gradually
over a number of years, in part by the authors (Sutton, 1990, 1991a, 1991b;
Barto, Bradtke, and Singh, 1991, 1995; Sutton and Pinette, 1985; Sutton andBarto, 1981b); it has been strongly influenced by Agre and Chapman (1990; Agre
1988), Bertsekas and Tsitsiklis (1989), Singh (1993), and others. The authors
were also strongly influenced by psychological studies of latent learning (Tolman,
1932) and by psychological views of the nature of thought (e.g., Galanter andGerstenhaber, 1956; Craik, 1943; Campbell, 1960; Dennett, 1978). In Part
III of the book, Section 14.6 relates model-based and model-free methods topsychological theories of learning and behavior, and Section 15.11 discusses ideas
about how the brain might implement these types of methods.

8.2

The terms direct and indirect, which we use to describe di↵erent kinds ofreinforcement learning, are from the adaptive control literature (e.g., Goodwin
and Sin, 1984), where they are used to make the same kind of distinction. The
term system identification is used in adaptive control for what we call modellearning (e.g., Goodwin and Sin, 1984; Ljung and Söderstrom, 1983; Young,1984). The Dyna architecture is due to Sutton (1990), and the results in thisand the next section are based on results reported there. Barto and Singh(1990) consider some of the issues in comparing direct and indirect reinforcement
learning methods. Early work extending Dyna to linear function approximation
was done by Sutton, Szepesvári, Geramifard, and Bowling (2008) and by Parr,
Li, Taylor, Painter-Wakefield, and Littman (2008).

8.3

There have been several works with model-based reinforcement learning that takethe idea of exploration bonuses and optimistic initialization to its logical extreme,in which all incompletely explored choices are assumed maximally rewarding
and optimal paths are computed to test them. The E3 algorithm of Kearns andSingh (2002) and the R-max algorithm of Brafman and Tennenholtz (2003) areguaranteed to find a near-optimal solution in time polynomial in the numberof states and actions. This is usually too slow for practical algorithms but isprobably the best that can be done in the worst case.

8.4

Prioritized sweeping was developed simultaneously and independently by Moore
and Atkeson (1993) and Peng and Williams (1993). The results in the box onpage 170 are due to Peng and Williams (1993). The results in the box on page 171
are due to Moore and Atkeson. Key subsequent work in this area includes thatby McMahan and Gordon (2005) and by van Seijen and Sutton (2013).

8.5This section was strongly influenced by the experiments of Singh (1993).

8.6–7 Trajectory sampling has implicitly been a part of reinforcement learning fromthe outset, but it was most explicitly emphasized by Barto, Bradtke, and Singh(1995) in their introduction of RTDP. They recognized that Korf’s (1990) learning

8.13. Summary of Part I: Dimensions

193

real-time A* (LRTA*) algorithm is an asynchronous DP algorithm that applies
to stochastic problems as well as the deterministic problems on which Korf
focused. Beyond LRTA*, RTDP includes the option of updating the values ofmany states in the time intervals between the execution of actions. Barto etal. (1995) proved the convergence result described here by combining Korf’s (1990)
convergence proof for LRTA* with the result of Bertsekas (1982) (also Bertsekas
and Tsitsiklis, 1989) ensuring convergence of asynchronous DP for stochastic
shortest path problems in the undiscounted case. Combining model-learningwith RTDP is called Adaptive RTDP, also presented by Barto et al. (1995) anddiscussed by Barto (2011).
8.9

For further reading on heuristic search, the reader is encouraged to consult texts
and surveys such as those by Russell and Norvig (2009) and Korf (1988). Pengand Williams (1993) explored a forward focusing of updates much as is suggested
in this section.

8.10

Abramson’s (1990) expected-outcome model is a rollout algorithm applied to twoperson games in which the play of both simulated players is random. He argued
that even with random play, it is a “powerful heuristic” that is “precise, accurate,easily estimable, efficiently calculable, and domain-independent.” Tesauro andGalperin (1997) demonstrated the e↵ectiveness of rollout algorithms for improving
the play of backgammon programs, adopting the term “rollout” from its use
in evaluating backgammon positions by playing out positions with di↵erent
randomly generating sequences of dice rolls. Bertsekas, Tsitsiklis, and Wu (1997)
examine rollout algorithms applied to combinatorial optimization problems, andBertsekas (2013) surveys their use in discrete deterministic optimization problems,
remarking that they are “often surprisingly e↵ective.”

8.11

The central ideas of MCTS were introduced by Coulom (2006) and by Kocsis
and Szepesvári (2006). They built upon previous research with Monte Carloplanning algorithms as reviewed by these authors. Browne, Powley, Whitehouse,
Lucas, Cowling, Rohlfshagen, Tavener, Perez, Samothrakis, and Colton (2012)is an excellent survey of MCTS methods and their applications. David Silver
contributed to the ideas and presentation in this section.

Part II:
Approximate Solution Methods

In the second part of the book we extend the tabular methods presented in the first partto apply to problems with arbitrarily large state spaces. In many of the tasks to which wewould like to apply reinforcement learning the state space is combinatorial and enormous;the number of possible camera images, for example, is much larger than the number ofatoms in the universe. In such cases we cannot expect to find an optimal policy or the
optimal value function even in the limit of infinite time and data; our goal instead is tofind a good approximate solution using limited computational resources. In this part ofthe book we explore such approximate solution methods.
The problem with large state spaces is not just the memory needed for large tables,
but the time and data needed to fill them accurately. In many of our target tasks, almost
every state encountered will never have been seen before. To make sensible decisions insuch states it is necessary to generalize from previous encounters with di↵erent statesthat are in some sense similar to the current one. In other words, the key issue is that ofgeneralization. How can experience with a limited subset of the state space be usefully
generalized to produce a good approximation over a much larger subset?
Fortunately, generalization from examples has already been extensively studied, andwe do not need to invent totally new methods for use in reinforcement learning. To someextent we need only combine reinforcement learning methods with existing generalization
methods. The kind of generalization we require is often called function approximation
because it takes examples from a desired function (e.g., a value function) and attempts
to generalize from them to construct an approximation of the entire function. Function
approximation is an instance of supervised learning, the primary topic studied in machinelearning, artificial neural networks, pattern recognition, and statistical curve fitting. In
theory, any of the methods studied in these fields can be used in the role of function
approximator within reinforcement learning algorithms, although in practice some fit
more easily into this role than others.
Reinforcement learning with function approximation involves a number of new issues
that do not normally arise in conventional supervised learning, such as nonstationarity,bootstrapping, and delayed targets. We introduce these and other issues successively overthe five chapters of this part. Initially we restrict attention to on-policy training, treatingin Chapter 9 the prediction case, in which the policy is given and only its value function
is approximated, and then in Chapter 10 the control case, in which an approximation tothe optimal policy is found. The challenging problem of o↵-policy learning with function
approximation is treated in Chapter 11. In each of these three chapters we will have

196Part II: Approximate Solution Methods

to return to first principles and re-examine the objectives of the learning to take into
account function approximation. Chapter 12 introduces and analyzes the algorithmicmechanism of eligibility traces, which dramatically improves the computational propertiesof multi-step reinforcement learning methods in many cases. The final chapter of thispart explores a di↵erent approach to control, policy-gradient methods, which approximate
the optimal policy directly and need never form an approximate value function (although
they may be much more efficient if they do approximate a value function as well thepolicy).

Chapter 9

On-policy Prediction with
Approximation
In this chapter, we begin our study of function approximation in reinforcement learningby considering its use in estimating the state-value function from on-policy data, that is,in approximating v⇡ from experience generated using a known policy ⇡. The novelty inthis chapter is that the approximate value function is represented not as a table but as aparameterized functional form with weight vector w 2 Rd . We will write v̂(s,w) ⇡ v⇡ (s)for the approximate value of state s given weight vector w. For example, v̂ might bea linear function in features of the state, with w the vector of feature weights. Moregenerally, v̂ might be the function computed by a multi-layer artificial neural network,with w the vector of connection weights in all the layers. By adjusting the weights, anyof a wide range of di↵erent functions can be implemented by the network. Or v̂ might be
the function computed by a decision tree, where w is all the numbers defining the split
points and leaf values of the tree. Typically, the number of weights (the dimensionality ofw) is much less than the number of states (d ⌧ |S|), and changing one weight changes theestimated value of many states. Consequently, when a single state is updated, the change
generalizes from that state to a↵ect the values of many other states. Such generalization
makes the learning potentially more powerful but also potentially more difficult to manage
and understand.
Perhaps surprisingly, extending reinforcement learning to function approximation also
makes it applicable to partially observable problems, in which the full state is not available
to the agent. If the parameterized function form for v̂ does not allow the estimatedvalue to depend on certain aspects of the state, then it is just as if those aspects areunobservable. In fact, all the theoretical results for methods using function approximation
presented in this part of the book apply equally well to cases of partial observability.
What function approximation can’t do, however, is augment the state representation
with memories of past observations. Some such possible further extensions are discussed
briefly in Section 17.3.

198

9.1

Chapter 9: On-policy Prediction with Approximation

Value-function Approximation

All of the prediction methods covered in this book have been described as updates to an
estimated value function that shift its value at particular states toward a “backed-up value,”or update target, for that state. Let us refer to an individual update by the notation s 7! u,where s is the state updated and u is the update target that s’s estimated value is shifted
toward. For example, the Monte Carlo update for value prediction is St 7! Gt , the TD(0)update is St 7! Rt+1 + v̂(St+1 ,wt ), and the n-step TD update is St 7! Gt:t+n . In the DP(dynamic programming) policy-evaluation update, s 7! E⇡[Rt+1 + v̂(St+1 ,wt ) | St = s],an arbitrary state s is updated, whereas in the other cases the state encountered in actual
experience, St , is updated.
It is natural to interpret each update as specifying an example of the desired input–
output behavior of the value function. In a sense, the update s 7! u means that theestimated value for state s should be more like the update target u. Up to now, theactual update has been trivial: the table entry for s’s estimated value has simply beenshifted a fraction of the way toward u, and the estimated values of all other states
were left unchanged. Now we permit arbitrarily complex and sophisticated methods toimplement the update, and updating at s generalizes so that the estimated values ofmany other states are changed as well. Machine learning methods that learn to mimic
input–output examples in this way are called supervised learning methods, and when the
outputs are numbers, like u, the process is often called function approximation. Function
approximation methods expect to receive examples of the desired input–output behaviorof the function they are trying to approximate. We use these methods for value prediction
simply by passing to them the s 7! u of each update as a training example. We theninterpret the approximate function they produce as an estimated value function.
Viewing each update as a conventional training example in this way enables us to use
any of a wide range of existing function approximation methods for value prediction. In
principle, we can use any method for supervised learning from examples, including artificial
neural networks, decision trees, and various kinds of multivariate regression. However,
not all function approximation methods are equally well suited for use in reinforcement
learning. The most sophisticated artificial neural network and statistical methods all
assume a static training set over which multiple passes are made. In reinforcement
learning, however, it is important that learning be able to occur online, while the agent
interacts with its environment or with a model of its environment. To do this requires
methods that are able to learn efficiently from incrementally acquired data. In addition,
reinforcement learning generally requires function approximation methods able to handle
nonstationary target functions (target functions that change over time). For example,
in control methods based on GPI (generalized policy iteration) we often seek to learnq⇡ while ⇡ changes. Even if the policy remains the same, the target values of training
examples are nonstationary if they are generated by bootstrapping methods (DP and TD
learning). Methods that cannot easily handle such nonstationarity are less suitable forreinforcement learning.

9.2. The Prediction Objective (VE)

9.2

199

The Prediction Objective (VE)

Up to now we have not specified an explicit objective for prediction. In the tabular casea continuous measure of prediction quality was not necessary because the learned valuefunction could come to equal the true value function exactly. Moreover, the learned
values at each state were decoupled—an update at one state a↵ected no other. But withgenuine approximation, an update at one state a↵ects many others, and it is not possible
to get the values of all states exactly correct. By assumption we have far more states
than weights, so making one state’s estimate more accurate invariably means making
others’ less accurate. We are obligated thenPto say which states we care most about. We
must specify a state distribution µ(s) 0, s µ(s) = 1, representing how much we careabout the error in each state s. By the error in a state s we mean the square of thedi↵erence between the approximate value v̂(s,w) and the true value v⇡ (s). Weighting
this over the state space by µ, we obtain a natural objective function, the mean squarevalue error, denoted VE:
hi2. X
VE(w) =µ(s) v⇡ (s) v̂(s,w) .
(9.1)
s2S

The square root of this measure, the root VE, gives a rough measure of how much theapproximate values di↵er from the true values and is often used in plots. Often µ(s) is
chosen to be the fraction of time spent in s. Under on-policy training this is called the
on-policy distribution; we focus entirely on this case in this chapter. In continuing tasks,
the on-policy distribution is the stationary distribution under ⇡.
The on-policy distribution in episodic tasks
In an episodic task, the on-policy distribution is a little di↵erent in that it depends
on how the initial states of episodes are chosen. Let h(s) denote the probability
that an episode begins in each state s, and let ⌘(s) denote the number of time
steps spent, on average, in state s in a single episode. Time is spent in a state s
if episodes start in s, or if transitions are made into s from a preceding state s̄ inwhich time is spent:XX⌘(s) = h(s) +⌘(s̄)⇡(a|s̄)p(s| s̄, a), for all s 2 S.
(9.2)
s̄

a

This system of equations can be solved for the expected number of visits ⌘(s). The
on-policy distribution is then the fraction of time spent in each state normalized tosum to one:µ(s) = P⌘(s), for all s 2 S.
0s0 ⌘(s )(9.3)

This is the natural choice without discounting. If there is discounting ( < 1) itshould be treated as a form of termination, which can be done simply by including
a factor of in the second term of (9.2).

200

Chapter 9: On-policy Prediction with Approximation

The two cases, continuing and episodic, behave similarly, but with approximation theymust be treated separately in formal analyses, as we will see repeatedly in this part ofthe book. This completes the specification of the learning objective.
It is not completely clear that the VE is the right performance objective for reinforcement learning. Remember that our ultimate purpose—the reason we are learninga value function—is to find a better policy. The best value function for this purpose is
not necessarily the best for minimizing VE. Nevertheless, it is not yet clear what a more
useful alternative goal for value prediction might be. For now, we will focus on VE.
An ideal goal in terms of VE would be to find a global optimum, a weight vector w⇤
for which VE(w⇤ )  VE(w) for all possible w. Reaching this goal is sometimes possible
for simple function approximators such as linear ones, but is rarely possible for complex
function approximators such as artificial neural networks and decision trees. Short ofthis, complex function approximators may seek to converge instead to a local optimum,a weight vector w⇤ for which VE(w⇤ )  VE(w) for all w in some neighborhood of w⇤ .
Although this guarantee is only slightly reassuring, it is typically the best that can besaid for nonlinear function approximators, and often it is enough. Still, for many cases ofinterest in reinforcement learning there is no guarantee of convergence to an optimum, oreven to within a bounded distance of an optimum. Some methods may in fact diverge,with their VE approaching infinity in the limit.
In the last two sections we outlined a framework for combining a wide range ofreinforcement learning methods for value prediction with a wide range of function
approximation methods, using the updates of the former to generate training examples
for the latter. We also described a VE performance measure which these methods may
aspire to minimize. The range of possible function approximation methods is far toolarge to cover all, and anyway too little is known about most of them to make a reliableevaluation or recommendation. Of necessity, we consider only a few possibilities. In
the rest of this chapter we focus on function approximation methods based on gradientprinciples, and on linear gradient-descent methods in particular. We focus on thesemethods in part because we consider them to be particularly promising and because they
reveal key theoretical issues, but also because they are simple and our space is limited.

9.3

Stochastic-gradient and Semi-gradient Methods

We now develop in detail one class of learning methods for function approximation invalue prediction, those based on stochastic gradient descent (SGD). SGD methods areamong the most widely used of all function approximation methods and are particularly
well suited to online reinforcement learning.
In gradient-descent methods, the weight vector is a column vector with a fixed number.
of real valued components, w = (w1 , w2 , . . . , wd )> ,1 and the approximate value functionv̂(s,w) is a di↵erentiable function of w for all s 2 S. We will be updating w at each ofa series of discrete time steps, t = 0, 1, 2, 3, . . ., so we will need a notation wt for the
1 The > denotes transpose, needed here to turn the horizontal row vector in the text into a verticalcolumn vector; in this book vectors are generally taken to be column vectors unless explicitly written out
horizontally or transposed.

9.3. Stochastic-gradient and Semi-gradient Methods

201

weight vector at each step. For now, let us assume that, on each step, we observe a new
example St 7! v⇡ (St ) consisting of a (possibly randomly selected) state St and its true
value under the policy. These states might be successive states from an interaction withthe environment, but for now we do not assume so. Even though we are given the exact,
correct values, v⇡ (St ) for each St , there is still a difficult problem because our function
approximator has limited resources and thus limited resolution. In particular, there is
generally no w that gets all the states, or even all the examples, exactly correct. In
addition, we must generalize to all the other states that have not appeared in examples.
We assume that states appear in examples with the same distribution, µ, over which
we are trying to minimize the VE as given by (9.1). A good strategy in this case isto try to minimize error on the observed examples. Stochastic gradient-descent (SGD)
methods do this by adjusting the weight vector after each example by a small amount in
the direction that would most reduce the error on that example:.
wt+1 = wth1↵r v⇡ (St )2h= wt + ↵ v⇡ (St )i2v̂(St ,wt )iv̂(St ,wt ) rv̂(St ,wt ),

(9.4)
(9.5)

where ↵ is a positive step-size parameter, and rf (w), for any scalar expression f (w)that is a function of a vector (here w), denotes the column vector of partial derivativesof the expression with respect to the components of the vector:.
rf (w) =✓@f (w) @f (w)@f (w)
,
,...,
@w1
@w2
@wd

◆>

.

(9.6)

This derivative vector is the gradient of f with respect to w. SGD methods are “gradientdescent” methods because the overall step in wt is proportional to the negative gradientof the example’s squared error (9.4). This is the direction in which the error falls most
rapidly. Gradient descent methods are called “stochastic” when the update is done, as
here, on only a single example, which might have been selected stochastically. Over many
examples, making small steps, the overall e↵ect is to minimize an average performancemeasure such as the VE.
It may not be immediately apparent why SGD takes only a small step in the directionof the gradient. Could we not move all the way in this direction and completely eliminate
the error on the example? In many cases this could be done, but usually it is not desirable.
Remember that we do not seek or expect to find a value function that has zero error for
all states, but only an approximation that balances the errors in di↵erent states. If wecompletely corrected each example in one step, then we would not find such a balance.
In fact, the convergence results for SGD methods assume that ↵ decreases over time. If
it decreases in such a way as to satisfy the standard stochastic approximation conditions(2.7), then the SGD method (9.5) is guaranteed to converge to a local optimum.
We turn now to the case in which the target output, here denoted Ut 2 R, of the tth
training example, St 7! Ut , is not the true value, v⇡ (St ), but some, possibly random,
approximation to it. For example, Ut might be a noise-corrupted version of v⇡ (St ), or it
might be one of the bootstrapping targets using v̂ mentioned in the previous section. In

202

Chapter 9: On-policy Prediction with Approximationthese cases we cannot perform the exact update (9.5) because v⇡ (St ) is unknown, but
we can approximate it by substituting Ut in place of v⇡ (St ). This yields the following
general SGD method for state-value prediction:hi.
wt+1 = wt + ↵ Ut v̂(St ,wt ) rv̂(St ,wt ).
(9.7)

If Ut is an unbiased estimate, that is, if E[Ut |St = s] = v⇡ (s), for each t, then wt isguaranteed to converge to a local optimum under the usual stochastic approximation
conditions (2.7) for decreasing ↵.
For example, suppose the states in the examples are the states generated by interaction(or simulated interaction) with the environment using policy ⇡. Because the true value of.
a state is the expected value of the return following it, the Monte Carlo target Ut = Gt is
by definition an unbiased estimate of v⇡ (St ). With this choice, the general SGD method(9.7) converges to a locally optimal approximation to v⇡ (St ). Thus, the gradient-descentversion of Monte Carlo state-value prediction is guaranteed to find a locally optimal
solution. Pseudocode for a complete algorithm is shown in the box below.
Gradient Monte Carlo Algorithm for Estimating v̂ ⇡ v⇡
Input: the policy ⇡ to be evaluatedInput: a di↵erentiable function v̂ : S ⇥ Rd ! R
Algorithm parameter: step size ↵ > 0
Initialize value-function weights w 2 Rd arbitrarily (e.g., w = 0)
Loop forever (for each episode):
Generate an episode S0 , A0 , R1 , S1 , A1 , . . . , RT , ST using ⇡Loop for each stepof episode,⇤ t = 0, 1, . . . , T 1:⇥w
w + ↵ Gt v̂(St ,w) rv̂(St ,w)

One does not obtain the same guarantees if a bootstrapping estimate of v⇡ (St ) is usedas the target
Ut in (9.7). Bootstrapping targets such as n-step returns Gt:t+n or the DP
Ptarget a,s0 ,r ⇡(a|St )p(s0 , r |St , a)[r + v̂(s0 ,wt )] all depend on the current value of theweight vector wt , which implies that they will be biased and that they will not produce atrue gradient-descent method. One way to look at this is that the key step from (9.4)
to (9.5) relies on the target being independent of wt . This step would not be valid if
a bootstrapping estimate were used in place of v⇡ (St ). Bootstrapping methods are notin fact instances of true gradient descent (Barnard, 1993). They take into account thee↵ect of changing the weight vector wt on the estimate, but ignore its e↵ect on the target.
They include only a part of the gradient and, accordingly, we call them semi-gradientmethods.
Although semi-gradient (bootstrapping) methods do not converge as robustly as
gradient methods, they do converge reliably in important cases such as the linear casediscussed in the next section. Moreover, they o↵er important advantages that make themoften clearly preferred. One reason for this is that they typically enable significantly fasterlearning, as we have seen in Chapters 6 and 7. Another is that they enable learning to9.3. Stochastic-gradient and Semi-gradient Methods

203

be continual and online, without waiting for the end of an episode. This enables them tobe used on continuing problems and provides computational advantages. A prototypical.
semi-gradient method is semi-gradient TD(0), which uses Ut = Rt+1 + v̂(St+1 ,w) as its
target. Complete pseudocode for this method is given in the box below.
Semi-gradient TD(0) for estimating v̂ ⇡ v⇡
Input: the policy ⇡ to be evaluatedInput: a di↵erentiable function v̂ : S+ ⇥ Rd ! R such that v̂(terminal,·) = 0
Algorithm parameter: step size ↵ > 0
Initialize value-function weights w 2 Rd arbitrarily (e.g., w = 0)
Loop for each episode:
Initialize S
Loop for each step of episode:
Choose A ⇠ ⇡(·|S)
Take action ⇥A, observe R, S 0⇤w
w + ↵ R + v̂(S 0 ,w) v̂(S,w) rv̂(S,w)
S
S0
until S is terminal
State aggregation is a simple form of generalizing function approximation in whichstates are grouped together, with one estimated value (one component of the weightvector w) for each group. The value of a state is estimated as its group’s component,
and when the state is updated, that component alone is updated. State aggregation
is a special case of SGD (9.7) in which the gradient, rv̂(St ,wt ), is 1 for St ’s group’s
component and 0 for the other components.
Example 9.1: State Aggregation on the 1000-state Random Walk Consider a1000-state version of the random walk task (Examples 6.2 and 7.1 on pages 125 and144). The states are numbered from 1 to 1000, left to right, and all episodes begin near
the center, in state 500. State transitions are from the current state to one of the 100
neighboring states to its left, or to one of the 100 neighboring states to its right, all withequal probability. Of course, if the current state is near an edge, then there may be fewer
than 100 neighbors on that side of it. In this case, all the probability that would havegone into those missing neighbors goes into the probability of terminating on that side
(thus, state 1 has a 0.5 chance of terminating on the left, and state 950 has a 0.25 chanceof terminating on the right). As usual, termination on the left produces a reward of
1, and termination on the right produces a reward of +1. All other transitions have areward of zero. We use this task as a running example throughout this section.
Figure 9.1 shows the true value function v⇡ for this task. It is nearly a straight line,
curving very slightly toward the horizontal for the last 100 states at each end. Also shownis the final approximate value function learned by the gradient Monte-Carlo algorithmwith state aggregation after 100,000 episodes with a step size of ↵ = 2 ⇥ 10 5 . For thestate aggregation, the 1000 states were partitioned into 10 groups of 100 states each (i.e.,
states 1–100 were one group, states 101–200 were another, and so on). The staircase e↵ect

204

Chapter 9: On-policy Prediction with Approximation

1

0.0137

True
value v⇡Value Approximate
0 MC value v̂scale

Distribution
scaleState distribution µ

-1

1

State

1000

0.00170

Figure 9.1: Function approximation by state aggregation on the 1000-state random walk task,using the gradient Monte Carlo algorithm (page 202).

shown in the figure is typical of state aggregation; within each group, the approximate
value is constant, and it changes abruptly from one group to the next. These approximate
values are close to the global minimum of the VE (9.1).
Some of the details of the approximate values are best appreciated by reference tothe state distribution µ for this task, shown in the lower portion of the figure with aright-side scale. State 500, in the center, is the first state of every episode, but is rarely
visited again. On average, about 1.37% of the time steps are spent in the start state.
The states reachable in one step from the start state are the second most visited, with
about 0.17% of the time steps being spent in each of them. From there µ falls o↵ almost
linearly, reaching about 0.0147% at the extreme states 1 and 1000. The most visiblee↵ect of the distribution is on the leftmost groups, whose values are clearly shifted higher
than the unweighted average of the true values of states within the group, and on therightmost groups, whose values are clearly shifted lower. This is due to the states inthese areas having the greatest asymmetry in their weightings by µ. For example, in theleftmost group, state 100 is weighted more than 3 times more strongly than state 1. Thus
the estimate for the group is biased toward the true value of state 100, which is higher
than the true value of state 1.

9.4

Linear Methods

One of the most important special cases of function approximation is that in which theapproximate function, v̂(·,w), is a linear function of the weight vector, w. Corresponding.
to every state s, there is a real-valued vector x(s) = (x1 (s), x2 (s), . . . , xd (s))> , with thesame number of components as w. Linear methods approximate the state-value function

9.4. Linear Methods

205

by the inner product between w and x(s):d.
. Xv̂(s,w) = w> x(s) =wi xi (s).

(9.8)

i=1

In this case the approximate value function is said to be linear in the weights, or simply
linear.
The vector x(s) is called a feature vector representing state s. Each component xi (s)of x(s) is the value of a function xi : S ! R. We think of a feature as the entirety of oneof these functions, and we call its value for a state s a feature of s. For linear methods,
features are basis functions because they form a linear basis for the set of approximate
functions. Constructing d-dimensional feature vectors to represent states is the same as
selecting a set of d basis functions. Features may be defined in many di↵erent ways; wecover a few possibilities in the next sections.
It is natural to use SGD updates with linear function approximation. The gradient of
the approximate value function with respect to w in this case isrv̂(s,w) = x(s).
Thus, in the linear case the general SGD update (9.7) reduces to a particularly simple
form:
h
i
.
wt+1 = wt + ↵ Ut v̂(St ,wt ) x(St ).

Because it is so simple, the linear SGD case is one of the most favorable for mathematicalanalysis. Almost all useful convergence results for learning systems of all kinds are forlinear (or simpler) function approximation methods.
In particular, in the linear case there is only one optimum (or, in degenerate cases,
one set of equally good optima), and thus any method that is guaranteed to converge toor near a local optimum is automatically guaranteed to converge to or near the global
optimum. For example, the gradient Monte Carlo algorithm presented in the previous
section converges to the global optimum of the VE under linear function approximation
if ↵ is reduced over time according to the usual conditions.
The semi-gradient TD(0) algorithm presented in the previous section also converges
under linear function approximation, but this does not follow from general results on
SGD; a separate theorem is necessary. The weight vector converged to is also not the
global optimum, but rather a point near the local optimum. It is useful to consider thisimportant case in more detail, specifically for the continuing case. The update at eachtime t is⇣⌘.
wt+1 = wt + ↵ Rt+1 + wt> xt+1 wt> xt xt
(9.9)
⇣
⌘
>
= wt + ↵ Rt+1 xt xt xt
xt+1 wt ,where here we have used the notational shorthand xt = x(St ). Once the system hasreached steady state, for any given wt , the expected next weight vector can be written
E[wt+1 |wt ] = wt + ↵(b

Awt ),(9.10)

206

Chapter 9: On-policy Prediction with Approximation

where.
b = E[Rt+1 xt ] 2 Rd

and

h
.
A = E xt xt

xt+1

>

i

2 Rd⇥d

(9.11)

From (9.10) it is clear that, if the system converges, it must converge to the weight vector
wTD at whichb

AwTD = 0)b = AwTD.
wTD = A 1 b.

)

(9.12)

This quantity is called the TD fixed point. In fact linear semi-gradient TD(0) converges
to this point. Some of the theory proving its convergence, and the existence of the inverse
above, is given in the box.
Proof of Convergence of Linear TD(0)What properties assure convergence of the linear TD(0) algorithm (9.9)? Some
insight can be gained by rewriting (9.10) as
E[wt+1 |wt ] = (I↵A)wt + ↵b.

(9.13)

Note that the matrix A multiplies the weight vector wt and not b; only A is
important to convergence. To develop intuition, consider the special case in which
A is a diagonal matrix. If any of the diagonal elements are negative, then thecorresponding diagonal element of I ↵A will be greater than one, and thecorresponding component of wt will be amplified, which will lead to divergence if
continued. On the other hand, if the diagonal elements of A are all positive, then↵ can be chosen smaller than one over the largest of them, such that I ↵A is
diagonal with all diagonal elements between 0 and 1. In this case the first termof the update tends to shrink wt , and stability is assured. In general, wt will be
reduced toward zero whenever A is positive definite, meaning y > Ay > 0 for any
real vector y 6= 0. Positive definiteness also ensures that the inverse A 1 exists.
For linear TD(0), in the continuing case with < 1, the A matrix (9.11) can be
written
X
X
X
>
A=
µ(s)
⇡(a|s)p(r, s0 |s, a)x(s) x(s)x(s0 )
s

=

X

µ(s)

s0

s

=

X

s
>

r,s0

a

X

p(s0 |s)x(s) x(s)✓µ(s)x(s) x(s)= X D(I

P)X,

X
s0

x(s0 )

00p(s |s)x(s )>◆>where µ(s) is the stationary distribution under ⇡, p(s0 |s) is the probability oftransition from s to s0 under policy ⇡, P is the |S| ⇥ |S| matrix of these probabilities,

9.4. Linear Methods

207

D is the |S| ⇥ |S| diagonal matrix with the µ(s) on its diagonal, and X is the |S| ⇥ dmatrix with x(s) as its rows. From here it is clear that the inner matrix D(IP)
is key to determining the positive definiteness of A.
For a key matrix of this form, positive definiteness is assured if all of its columns
sum to a nonnegative number. This was shown by Sutton (1988, p. 27) basedon two previously established theorems. One theorem says that any matrix M
is positive definite if and only if the symmetric matrix S = M + M> is positive
definite (Sutton 1988, appendix). The second theorem says that any symmetricreal matrix S is positive definite if all of its diagonal entries are positive and greaterthan the sum of the absolute values of the corresponding o↵-diagonal entries (Varga
1962, p. 23). For our key matrix, D(IP), the diagonal entries are positive
and the o↵-diagonal entries are negative, so all we have to show is that each row
sum plus the corresponding column sum is positive. The row sums are all positive
because P is a stochastic matrix and < 1. Thus it only remains to show thatthe column sums are nonnegative. Note that the row vector of the column sumsof any matrix M can be written as 1> M, where 1 is the column vector with allcomponents equal to 1. Let µ denote the |S|-vector of the µ(s), where µ = P> µ by
virtue of µ being the stationary distribution. The column sums of our key matrix,
then, are:1> D(I

P) = µ> (I

P)

= µ>

µ> P

= µ>

µ>

= (1

>

(because µ is the stationary distribution))µ ,all components of which are positive. Thus, the key matrix and its A matrix
are positive definite, and on-policy TD(0) is stable. (Additional conditions and aschedule for reducing ↵ over time are needed to prove convergence with probabilityone.)
At the TD fixed point, it has also been proven (in the continuing case) that the VE is
within a bounded expansion of the lowest possible error:
VE(wTD ) 11

min VE(w).
w(9.14)

That is, the asymptotic error of the TD method is no more than 1 1 times the smallest
possible error, that attained in the limit by the Monte Carlo method. Because is often
near one, this expansion factor can be quite large, so there is substantial potential loss inasymptotic performance with the TD method. On the other hand, recall that the TD
methods are often of vastly reduced variance compared to Monte Carlo methods, andthus faster, as we saw in Chapters 6 and 7. Which method will be best depends on thenature of the approximation and problem, and on how long learning continues.

208

Chapter 9: On-policy Prediction with Approximation

A bound analogous to (9.14) applies to other on-policy
bootstrapping
P methods as well.
. P
For example, linear semi-gradient DP (Eq. 9.7 with Ut = a ⇡(a|St ) s0 ,r p(s0 , r |St , a)[r+v̂(s0 ,wt )]) with updates according to the on-policy distribution will also converge to
the TD fixed point. One-step semi-gradient action-value methods, such as semi-gradientSarsa(0) covered in the next chapter converge to an analogous fixed point and an analogous
bound. For episodic tasks, there is a slightly di↵erent but related bound (see Bertsekas
and Tsitsiklis, 1996). There are also a few technical conditions on the rewards, features,and decrease in the step-size parameter, which we have omitted here. The full details
can be found in the original paper (Tsitsiklis and Van Roy, 1997).
Critical to these convergence results is that states are updated according to the onpolicy distribution. For other update distributions, bootstrapping methods using function
approximation may actually diverge to infinity. Examples of this and a discussion of
possible solution methods are given in Chapter 11.
Example 9.2: Bootstrapping on the 1000-state Random Walk State aggregation
is a special case of linear function approximation, so let’s return to the 1000-state random
walk to illustrate some of the observations made in this chapter. The left panel of
Figure 9.2 shows the final value function learned by the semi-gradient TD(0) algorithm(page 203) using the same state aggregation as in Example 9.1. We see that the nearasymptotic TD approximation is indeed farther from the true values than the Monte
Carlo approximation shown in Figure 9.1.
Nevertheless, TD methods retain large potential advantages in learning rate, andgeneralize Monte Carlo methods, as we investigated fully with n-step TD methods in
Chapter 7. The right panel of Figure 9.2 shows results with an n-step semi-gradient
TD method using state aggregation on the 1000-state random walk that are strikingly
similar to those we obtained earlier with tabular methods and the 19-state random
walk (Figure 7.2). To obtain such quantitatively similar results we switched the state
aggregation to 20 groups of 50 states each. The 20 groups were then quantitatively close
256
512

0.55
1

True
value v⇡

n=64
0.5

n=32

Average
0.45
RMS error
over 1000 states 0.4
and first 10
0.35
episodesApproximate
TD value v̂

0

n=1
n=16

0.3

-1

128

n=8

n=2

n=4

0.25
1

State

1000

0

0.2

0.4

0.6

0.8

1

↵

Figure 9.2: Bootstrapping with state aggregation on the 1000-state random walk task. Left:
Asymptotic values of semi-gradient TD are worse than the asymptotic Monte Carlo values in
Figure 9.1. Right: Performance of n-step methods with state-aggregation are strikingly similar
to those with tabular representations (cf. Figure 7.2). These data are averages over 100 runs.

9.4. Linear Methods

209

to the 19 states of the tabular problem. In particular, recall that state transitions were
up to 100 states to the left or right. A typical transition would then be of 50 states tothe right or left, which is quantitatively analogous to the single-state state transitions of
the 19-state tabular system. To complete the match, we use here the same performancemeasure—an unweighted average of the RMS error over all states and over the first
10 episodes—rather than a VE objective as is otherwise more appropriate when using
function approximation.
The semi-gradient n-step TD algorithm used in the example above is the naturalextension of the tabular n-step TD algorithm presented in Chapter 7 to semi-gradientfunction approximation. Pseudocode is given in the box below.
n-step semi-gradient TD for estimating v̂ ⇡ v⇡
Input: the policy ⇡ to be evaluatedInput: a di↵erentiable function v̂ : S+ ⇥ Rd ! R such that v̂(terminal,·) = 0
Algorithm parameters: step size ↵ > 0, a positive integer n
Initialize value-function weights w arbitrarily (e.g., w = 0)
All store and access operations (St and Rt ) can take their index mod n + 1
Loop for each episode:
Initialize and store S0 6= terminal
T
1
Loop for t = 0, 1, 2, . . . :| If t < T , then:|Take an action according to ⇡(·|St )|Observe and store the next reward as Rt+1 and the next state as St+1|
If St+1 is terminal, then Tt+1| ⌧t n + 1 (⌧ is the time whose state’s estimate is being updated)| If ⌧ 0:
Pmin(⌧ +n,T ) i ⌧ 1|
GRi
i=⌧ +1|
If ⌧ + n < T , then: G
G + n v̂(S⌧ +n ,w)(G⌧ :⌧ +n )|w
w + ↵ [G v̂(S⌧ ,w)] rv̂(S⌧ ,w)
Until ⌧ = T 1
The key equation of this algorithm, analogous to (7.2), is.
wt+n = wt+n 1 + ↵ [Gt:t+nv̂(St ,wt+n 1 )] rv̂(St ,wt+n 1 ),0  t < T, (9.15)

where the n-step return is generalized from (7.1) to.
Gt:t+n = Rt+1 + Rt+2 + · · · +n 1

Rt+n +

nv̂(St+n ,wt+n 1 ), 0  t  T

n. (9.16)

Exercise 9.1 Show that tabular methods such as presented in Part I of this book are a
special case of linear function approximation. What would the feature vectors be?
⇤

210

9.5

Chapter 9: On-policy Prediction with Approximation

Feature Construction for Linear Methods

Linear methods are interesting because of their convergence guarantees, but also because
in practice they can be very efficient in terms of both data and computation. Whether or
not this is so depends critically on how the states are represented in terms of features,
which we investigate in this large section. Choosing features appropriate to the task is
an important way of adding prior domain knowledge to reinforcement learning systems.
Intuitively, the features should correspond to the aspects of the state space along which
generalization may be appropriate. If we are valuing geometric objects, for example,
we might want to have features for each possible shape, color, size, or function. If weare valuing states of a mobile robot, then we might want to have features for locations,degrees of remaining battery power, recent sonar readings, and so on.
A limitation of the linear form is that it cannot take into account any interactions
between features, such as the presence of feature i being good only in the absence offeature j. For example, in the pole-balancing task (Example 3.4), high angular velocity
can be either good or bad depending on the angle. If the angle is high, then high angularvelocity means an imminent danger of falling—a bad state—whereas if the angle is low,then high angular velocity means the pole is righting itself—a good state. A linear valuefunction could not represent this if its features coded separately for the angle and theangular velocity. It needs instead, or in addition, features for combinations of these twounderlying state dimensions. In the following subsections we consider a variety of generalways of doing this.

9.5.1

Polynomials

The states of many problems are initially expressed as numbers, such as positions andvelocities in the pole-balancing task (Example 3.4), the number of cars in each lot in theJack’s car rental problem (Example 4.2), or the gambler’s capital in the gambler problem(Example 4.3). In these types of problems, function approximation for reinforcement
learning has much in common with the familiar tasks of interpolation and regression.
Various families of features commonly used for interpolation and regression can also be
used in reinforcement learning. Polynomials make up one of the simplest families offeatures used for interpolation and regression. While the basic polynomial features we
discuss here do not work as well as other types of features in reinforcement learning, they
serve as a good introduction because they are simple and familiar.
As an example, suppose a reinforcement learning problem has states with two numericaldimensions. For a single representative state s, let its two numbers be s1 2 R and s2 2 R.
You might choose to represent s simply by its two state dimensions, so that x(s) =(s1 , s2 )> , but then you would not be able to take into account any interactions between
these dimensions. In addition, if both s1 and s2 were zero, then the approximate value
would have to also be zero. Both limitations can be overcome by instead representing s by
the four-dimensional feature vector x(s) = (1, s1 , s2 , s1 s2 )> . The initial 1 feature allows
the representation of affine functions in the original state numbers, and the final productfeature, s1 s2 , enables interactions to be taken into account. Or you might choose to use
higher-dimensional feature vectors like x(s) = (1, s1 , s2 , s1 s2 , s21 , s22 , s1 s22 , s21 s2 , s21 s22 )> to9.5. Feature Construction for Linear Methods

211

take more complex interactions into account. Such feature vectors enable approximations
as arbitrary quadratic functions of the state numbers—even though the approximation isstill linear in the weights that have to be learned. Generalizing this example from two
to k numbers, we can represent highly-complex interactions among a problem’s state
dimensions:Suppose each state s corresponds to k numbers, s1 , s2 , ..., sk , with each si 2 R.
For this k-dimensional state space, each order-n polynomial-basis feature xi can be
written ascxi (s) = ⇧kj=1 sji,j ,(9.17)

where each ci,j is an integer in the set {0, 1, . . . , n} for an integer n 0. Thesefeatures make up the order-n polynomial basis for dimension k, which contains(n + 1)k di↵erent features.
Higher-order polynomial bases allow for more accurate approximations of more complicated functions. But because the number of features in an order-n polynomial basis grows
exponentially with the dimension k of the natural state space (if n > 0), it is generally
necessary to select a subset of them for function approximation. This can be done using
prior beliefs about the nature of the function to be approximated, and some automatedselection methods developed for polynomial regression can be adapted to deal with theincremental and nonstationary nature of reinforcement learning.
Exercise 9.2 Why does (9.17) define (n + 1)k distinct features for dimension k?

⇤

Exercise 9.3 What n and ci,j produce the feature vectors x(s) = (1, s1 , s2 , s1 s2 , s21 , s22 ,s1 s22 , s21 s2 , s21 s22 )> ?
⇤

9.5.2

Fourier Basis

Another linear function approximation method is based on the time-honored Fourier
series, which expresses periodic functions as weighted sums of sine and cosine basis
functions (features) of di↵erent frequencies. (A function f is periodic if f (x) = f (x + ⌧ )for all x and some period ⌧ .) The Fourier series and the more general Fourier transformare widely used in applied sciences in part because if a function to be approximated is
known, then the basis function weights are given by simple formulae and, further, withenough basis functions essentially any function can be approximated as accurately as
desired. In reinforcement learning, where the functions to be approximated are unknown,
Fourier basis functions are of interest because they are easy to use and can perform wellin a range of reinforcement learning problems.
First consider the one-dimensional case. The usual Fourier series representation of afunction of one dimension having period ⌧ represents the function as a linear combinationof sine and cosine functions that are each periodic with periods that evenly divide ⌧ (inother words, whose frequencies are integer multiples of a fundamental frequency 1/⌧ ).
But if you are interested in approximating an aperiodic function defined over a bounded

212

Chapter 9: On-policy Prediction with Approximationinterval, then you can use these Fourier basis features with ⌧ set to the length of theinterval. The function of interest is then just one period of the periodic linear combinationof the sine and cosine features.
Furthermore, if you set ⌧ to twice the length of the interval of interest and restrict
attention to the approximation over the half interval [0, ⌧ /2], then you can use just thecosine features. This is possible because you can represent any even function, that is,any function that is symmetric about the origin, with just the cosine basis. So any
function over the half-period [0, ⌧ /2] can be approximated as closely as desired withenough cosine features. (Saying “any function” is not exactly correct because the function
has to be mathematically well-behaved, but we skip this technicality here.) Alternatively,
it is possible to use just sine features, linear combinations of which are always odd
functions, that is functions that are anti-symmetric about the origin. But it is generally
better to keep just the cosine features because “half-even” functions tend to be easier toapproximate than “half-odd” functions because the latter are often discontinuous at theorigin. Of course, this does not rule out using both sine and cosine features to approximate
over the interval [0, ⌧ /2], which might be advantageous in some circumstances.
Following this logic and letting ⌧ = 2 so that the features are defined over the half-⌧interval [0, 1], the one-dimensional order-n Fourier cosine basis consists of the n + 1
featuresxi (s) = cos(i⇡s), s 2 [0, 1],
for i = 0, . . . , n. Figure 9.3 shows one-dimensional Fourier cosine features xi , for i =1, 2, 3, 4; x0 is a constant function.
Univariate Fourier Basis Function k=1

11

Univariate Fourier Basis Function k=2

11

Univariate Fourier Basis Function k=3

11

0.8

0.8

0.8

0.6

0.6

0.6

0.6

0.4

0.4

0.4

0.4

0.2

0.2

0.2

0

0

0

0

−0.2

−0.2

−0.2

−0.2

−0.4

−0.4

−0.4

−0.4

−0.6

−0.6

−0.6

−0.6

−0.8

−0.8

−0.8

-1−1

-1−1

-1−1

00

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

11

0
0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

11

Univariate Fourier Basis Function k=4

11

0.8

0.2

−0.8

0
0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

11

-1−1

0
0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

11

Figure 9.3: One-dimensional Fourier cosine-basis features xi , i = 1, 2, 3, 4, for approximating
functions over the interval [0, 1]. After Konidaris et al. (2011).

This same reasoning applies to the Fourier cosine series approximation in the multidimensional case as described in the box below.
Suppose each state s corresponds to a vector of k numbers, s = (s1 , s2 , ..., sk )> ,with each si 2 [0, 1]. The ith feature in the order-n Fourier cosine basis can thenbe writtenxi (s) = cos ⇡ s> ci ,i

(9.18)= (ci1 , . . . , cik )> , with cij 2 {0, . . . , n} for j = 1, . . . , k and i = 1, . . . , (n+1)k .
kiwhere cThis defines a feature for each of the (n + 1) possible integer vectors c . The inner9.5. Feature Construction for Linear Methods

213

product s> ci has the e↵ect of assigning an integer in {0, . . . , n} to each dimensionof s. As in the one-dimensional case, this integer determines the feature’s frequency
along that dimension. The features can of course be shifted and scaled to suit thebounded state space of a particular application.
As an example, consider the k = 2 case in which s = (s1 , s2 )> , where each ci = (ci1 , ci2 )> .
Figure 9.4 shows a selection of six Fourier cosine features, each labeled by the vector cithat defines it (s1 is the horizontal axis and ci is shown as a row vector with the index iomitted). Any zero in c means the feature is constant along that state dimension. So if
c = (0, 0)> , the feature is constant over both dimensions; if c = (c1 , 0)> the feature isconstant over the second dimension and varies over the first with frequency dependingon c1 ; and similarly, for c = (0, c2 )> . When c = (c1 , c2 )> with neither cj = 0, thefeature varies along both dimensions and represents an interaction between the two statevariables. The values of c1 and c2 determine the frequency along each dimension, andtheir ratio gives the direction of the interaction.
= (0,
c=
(0,1)
1)>

11

00

c=
(1,0)
0)>
= (1,

1

0

00

11

cc =
(0, 5)
5)
= (1,

1

>

0

0

1

cc =
(2, 5)
5)
= (2,

1

0

>

1

0

1

c = (5, 2)>

1

0
0

c=
(1,1)
1)>
= (1,

1

0

0

1

0

Figure 9.4: A selection of six two-dimensional Fourier cosine features, each labeled by thevector ci that defines it (s1 is the horizontal axis, and ci is shown with the index i omitted).
After Konidaris et al. (2011).

When using Fourier cosine features with a learning algorithm such as (9.7), semigradient TD(0), or semi-gradient Sarsa, it may be helpful to use a di↵erent step-size
parameter for each feature. If ↵ is the basic step-size parameter, then Konidaris, Osentoski,
andpThomas (2011) suggest setting the step-size parameter for feature xi to ↵i =↵/ (ci1 )2 + · · · + (cik )2 (except when each cij = 0, in which case ↵i = ↵).

1

214

Chapter 9: On-policy Prediction with Approximation

Fourier cosine features with Sarsa can produce good performance compared to several
other collections of basis functions, including polynomial and radial basis functions. Not
surprisingly, however, Fourier features have trouble with discontinuities because it isdifficult to avoid “ringing” around points of discontinuity unless very high frequency basis
functions are included.
The number of features in the order-n Fourier basis grows exponentially with thedimension of the state space, but if that dimension is small enough (e.g., k  5), then
one can select n so that all of the order-n Fourier features can be used. This makes theselection of features more-or-less automatic. For high dimension state spaces, however, itis necessary to select a subset of these features. This can be done using prior beliefs about
the nature of the function to be approximated, and some automated selection methods
can be adapted to deal with the incremental and nonstationary nature of reinforcement
learning. An advantage of Fourier basis features in this regard is that it is easy to selectfeatures by setting the ci vectors to account for suspected interactions among the statevariables and by limiting the values in the cj vectors so that the approximation canfilter out high frequency components considered to be noise. On the other hand, because
Fourier features are non-zero over the entire state space (with the few zeros excepted),
they represent global properties of states, which can make it difficult to find good ways
to represent local properties.
Figure 9.5 shows learning curves comparing the Fourier and polynomial bases on the
1000-state random walk example. In general, we do not recommend using polynomials
for online learning.2
.4

.3

p
VE

averagedover 30 runs

.2

Polynomial basis.1

Fourier basis
0
0

5000

EpisodesFigure 9.5: Fourier basis vs polynomials on the 1000-state random walk. Shown are learning
curves for the gradient Monte Carlo method with Fourier and polynomial bases of order 5, 10,
and 20. The step-size parameters were roughly optimized for each case: ↵ = 0.0001 for thepolynomial basis and ↵ = 0.00005 for the Fourier basis. The performance measure (y-axis) is
the root mean square value error (9.1).
2 There are families of polynomials more complicated than those we have discussed, for example,di↵erent families of orthogonal polynomials, and these might work better, but at present there is little
experience with them in reinforcement learning.

9.5. Feature Construction for Linear Methods

9.5.3

215

Coarse Coding

Consider a task in which the natural representation of the state set is a continuous twodimensional space. One kind of representation forthis case is made up of features corresponding tos0
circles in state space, as shown to the right. If
the state is inside a circle, then the correspondingsfeature has the value 1 and is said to be present;
otherwise the feature is 0 and is said to be absent.
This kind of 1–0-valued feature is called a binaryfeature. Given a state, which binary features arepresent indicate within which circles the state lies,
and thus coarsely code for its location. Representing a state with features that overlap in this way(although they need not be circles or binary) is Figure 9.6: Coarse coding. Generalizaknown as coarse coding.
tion from state s to state s0 depends on
Assuming linear gradient-descent function ap- the number of their features whose recepproximation, consider the e↵ect of the size and tive fields (in this case, circles) overlap.
density of the circles. Corresponding to each cir- These states have one feature in common,cle is a single weight (a component of w) that is so there will be slight generalization bea↵ected by learning. If we train at one state, a tween them.
point in the space, then the weights of all circles
intersecting that state will be a↵ected. Thus, by (9.8), the approximate value function
will be a↵ected at all states within the union of the circles, with a greater e↵ect the morecircles a point has “in common” with the state, as shown in Figure 9.6. If the circles are
small, then the generalization will be over a short distance, as in Figure 9.7 (left), whereas
if they are large, it will be over a large distance, as in Figure 9.7 (middle). Moreover,a) Narrow generalizationb) Broad generalizationc) Asymmetric generalization

Figure 9.7: Generalization in linear function approximation methods is determined by thesizes and shapes of the features’ receptive fields. All three of these cases have roughly the same
number and density of features.

216

Chapter 9: On-policy Prediction with Approximation

the shape of the features will determine the nature of the generalization. For example, ifthey are not strictly circular, but are elongated in one direction, then generalization will
be similarly a↵ected, as in Figure 9.7 (right).
Features with large receptive fields give broad generalization, but might also seem tolimit the learned function to a coarse approximation, unable to make discriminations
much finer than the width of the receptive fields. Happily, this is not the case. Initial
generalization from one point to another is indeed controlled by the size and shape ofthe receptive fields, but acuity, the finest discrimination ultimately possible, is controlled
more by the total number of features.
Example 9.3: Coarseness of Coarse Coding This example illustrates the e↵ect onlearning of the size of the receptive fields in coarse coding. Linear function approximation
based on coarse coding and (9.7) was used to learn a one-dimensional square-wave function(shown at the top of Figure 9.8). The values of this function were used as the targets, Ut .
With just one dimension, the receptive fields were intervals rather than circles. Learning
was repeated with three di↵erent sizes of the intervals: narrow, medium, and broad, as
shown at the bottom of the figure. All three cases had the same density of features,
about 50 over the extent of the function being learned. Training examples were generated
uniformly at random over this extent. The step-size parameter was ↵ = 0.2
n , where n is
the number of features that were present at one time. Figure 9.8 shows the functions
learned in all three cases over the course of learning. Note that the width of the features
had a strong e↵ect early in learning. With broad features, the generalization tended to bebroad; with narrow features, only the close neighbors of each trained point were changed,
causing the function learned to be more bumpy. However, the final function learned wasa↵ected only slightly by the width of the features. Receptive field shape tends to have astrong e↵ect on generalization but little e↵ect on asymptotic solution quality.
desiredfunction#Examples

approximation

10
40
160
640
2560
10240

featurewidth

NarrowfeaturesMediumfeaturesBroadfeatures

Figure 9.8: Example of feature width’s strong e↵ect on initial generalization (first row) andweak e↵ect on asymptotic accuracy (last row).

9.5. Feature Construction for Linear Methods

9.5.4

217

Tile Coding

Tile coding is a form of coarse coding for multi-dimensional continuous spaces that is
flexible and computationally efficient. It may be the most practical feature representation
for modern sequential digital computers.
In tile coding the receptive fields of the features are grouped into partitions of the state
space. Each such partition is called a tiling, and each element of the partition is called atile. For example, the simplest tiling of a two-dimensional state space is a uniform gridsuch as that shown on the left side of Figure 9.9. The tiles or receptive field here aresquares rather than the circles in Figure 9.6. If just this single tiling were used, then thestate indicated by the white spot would be represented by the single feature whose tile
it falls within; generalization would be complete to all states within the same tile andnonexistent to states outside it. With just one tiling, we would not have coarse coding
but just a case of state aggregation.
Tiling 1

Continuous
2D state
space

Tiling 2
Tiling 3
Tiling 4

Point instate space
to berepresented

Four active
tiles/features
overlap the point
and are used torepresent it

Figure 9.9: Multiple, overlapping grid-tilings on a limited two-dimensional space. These tilings
are o↵set from one another by a uniform amount in each dimension.

To get the strengths of coarse coding requires overlapping receptive fields, and bydefinition the tiles of a partition do not overlap. To get true coarse coding with tile coding,multiple tilings are used, each o↵set by a fraction of a tile width. A simple case withfour tilings is shown on the right side of Figure 9.9. Every state, such as that indicatedby the white spot, falls in exactly one tile in each of the four tilings. These four tiles
correspond to four features that become active when the state occurs. Specifically, thefeature vector x(s) has one component for each tile in each tiling. In this example there
are 4 ⇥ 4 ⇥ 4 = 64 components, all of which will be 0 except for the four corresponding to
the tiles that s falls within. Figure 9.10 shows the advantage of multiple o↵set tilings(coarse coding) over a single tiling on the 1000-state random walk example.
An immediate practical advantage of tile coding is that, because it works with partitions,the overall number of features that are active at one time is the same for any state.
Exactly one feature is present in each tiling, so the total number of features present isalways the same as the number of tilings. This allows the step-size parameter, ↵, to

218

Chapter 9: On-policy Prediction with Approximation

.4

p
VE

averagedover 30 runs

.3

.2

State aggregation(one tiling).1

Tile coding (50 tilings)
00

Episodes5000

Figure 9.10: Why we use coarse coding. Shown are learning curves on the 1000-state random
walk example for the gradient Monte Carlo algorithm with a single tiling and with multiple
tilings. The space of 1000 states was treated as a single continuous dimension, covered with tiles
each 200 states wide. The multiple tilings were o↵set from each other by 4 states. The step-size
parameter was set so that the initial learning rate in the two cases was the same, ↵ = 0.0001 forthe single tiling and ↵ = 0.0001/50 for the 50 tilings.

be set in an easy, intuitive way. For example, choosing ↵ = n1 , where n is the numberof tilings, results in exact one-trial learning. If the example s 7! v is trained on, thenwhatever the prior estimate, v̂(s,wt ), the new estimate will be v̂(s,wt+1 ) = v. Usually
one wishes to change more slowly than this, to allow for generalization and stochastic
1
variation in target outputs. For example, one might choose ↵ = 10n, in which case theestimate for the trained state would move one-tenth of the way to the target in one
update, and neighboring states will be moved less, proportional to the number of tiles
they have in common.
Tile coding also gains computational advantages from its use of binary feature vectors.
Because each component is either 0 or 1, the weighted sum making up the approximate
value function (9.8) is almost trivial to compute. Rather than performing d multiplications
and additions, one simply computes the indices of the n ⌧ d active features and thenadds up the n corresponding components of the weight vector.
Generalization occurs to states other than the one trained if those states fall within
any of the same tiles, proportional to the number of tiles in common. Even the choice of
how to o↵set the tilings from each other a↵ects generalization. If they are o↵set uniformly
in each dimension, as they were in Figure 9.9, then di↵erent states can generalize inqualitatively di↵erent ways, as shown in the upper half of Figure 9.11. Each of the eightsubfigures show the pattern of generalization from a trained state to nearby points. In thisexample there are eight tilings, thus 64 subregions within a tile that generalize distinctly,
but all according to one of these eight patterns. Note how uniform o↵sets result in astrong e↵ect along the diagonal in many patterns. These artifacts can be avoided if thetilings are o↵set asymmetrically, as shown in the lower half of the figure. These lower
generalization patterns are better because they are all well centered on the trained statewith no obvious asymmetries.

9.5. Feature Construction for Linear Methods

219

Possible
generalizations
for uniformly
oﬀset tilings

Possible
generalizations
for asymmetricallyoﬀset tilings

Figure 9.11: Why tile asymmetrical o↵sets are preferred in tile coding. Shown is the strengthof generalization from a trained state, indicated by the small black plus, to nearby states, for thecase of eight tilings. If the tilings are uniformly o↵set (above), then there are diagonal artifacts
and substantial variations in the generalization, whereas with asymmetrically o↵set tilings thegeneralization is more spherical and homogeneous.

Tilings in all cases are o↵set from each other by a fraction of a tile width in eachdimension. If w denotes the tile width and n the number of tilings, then w
n is a fundamental
unit. Within small squares w
on
a
side,
all
statesactivatethesametiles,
have the samenfeature representation, and the same approximated value. If a state is moved by wnin any cartesian direction, the feature representation changes by one component/tile.
Uniformly o↵set tilings are o↵set from each other by exactly this unit distance. For a
two-dimensional space, we say that each tiling is o↵set by the displacement vector (1, 1),
meaning that it is o↵set from the previous tiling by w
n times this vector. In these terms,
the asymmetrically o↵set tilings shown in the lower part of Figure 9.11 are o↵set by adisplacement vector of (1, 3).
Extensive studies have been made of the e↵ect of di↵erent displacement vectors on thegeneralization of tile coding (Parks and Militzer, 1991; An, 1991; An, Miller and Parks,

220

Chapter 9: On-policy Prediction with Approximation

1991; Miller, An, Glanz and Carter, 1990), assessing their homegeneity and tendency
toward diagonal artifacts like those seen for the (1, 1) displacement vectors. Based on thiswork, Miller and Glanz (1996) recommend using displacement vectors consisting of thefirst odd integers. In particular, for a continuous space of dimension k, a good choice isto use the first odd integers (1, 3, 5, 7, . . . , 2k 1), with n (the number of tilings) set to an
integer power of 2 greater than or equal to 4k. This is what we have done to produce thetilings in the lower half of Figure 9.11, in which k = 2, n = 23 4k, and the displacement
vector is (1, 3). In a three-dimensional case, the first four tilings would be o↵set in total
from a base position by (0, 0, 0), (1, 3, 5), (2, 6, 10), and (3, 9, 15). Open-source softwarethat can efficiently make tilings like this for any k is readily available.
In choosing a tiling strategy, one has to pick the number of the tilings and the shape ofthe tiles. The number of tilings, along with the size of the tiles, determines the resolution
or fineness of the asymptotic approximation, as in general coarse coding and illustrated
in Figure 9.8. The shape of the tiles will determine the nature of generalization as in
Figure 9.7. Square tiles will generalize roughly equally in each dimension as indicated in
Figure 9.11 (lower). Tiles that are elongated along one dimension, such as the stripe tilingsin Figure 9.12 (middle), will promote generalization along that dimension. The tilings in
Figure 9.12 (middle) are also denser and thinner on the left, promoting discrimination
along the horizontal dimension at lower values along that dimension. The diagonal stripe
tiling in Figure 9.12 (right) will promote generalization along one diagonal. In higher
dimensions, axis-aligned stripes correspond to ignoring some of the dimensions in someof the tilings, that is, to hyperplanar slices. Irregular tilings such as shown in Figure 9.12
(left) are also possible, though rare in practice and beyond the standard software.

a) Irregularb) Log stripesc) Diagonal stripes

Figure 9.12: Tilings need not be grids. They can be arbitrarily shaped and non-uniform, whilestill in many cases being computationally efficient to compute.

In practice, it is often desirable to use di↵erent shaped tiles in di↵erent tilings. For
example, one might use some vertical stripe tilings and some horizontal stripe tilings.
This would encourage generalization along either dimension. However, with stripe tilings
alone it is not possible to learn that a particular conjunction of horizontal and verticalcoordinates has a distinctive value (whatever is learned for it will bleed into states with thesame horizontal and vertical coordinates). For this one needs the conjunctive rectangulartiles such as originally shown in Figure 9.9. With multiple tilings—some horizontal, somevertical, and some conjunctive—one can get everything: a preference for generalizing
along each dimension, yet the ability to learn specific values for conjunctions (see Sutton,

9.5. Feature Construction for Linear Methods

221

1996 for examples). The choice of tilings determines generalization, and until this choice
can be e↵ectively automated, it is important that tile coding enables the choice to bemade flexibly and in a way that makes sense to people.
Another useful trick for reducing memory requirements is hashing—a consistent pseudorandom collapsing of a large tiling into a much smaller set of tiles. Hashing produces
tiles consisting of noncontiguous, disjoint regions randomly spread throughout the state
space, but that still form an exhaustive partition. For example,
one tile might consist of the four subtiles shown to the right.
Through hashing, memory requirements are often reduced bylarge factors with little loss of performance. This is possible
because high resolution is needed in only a small fraction of theone
state space. Hashing frees us from the curse of dimensionality
tile
in the sense that memory requirements need not be exponential
in the number of dimensions, but need merely match the realdemands of the task. Open-source implementations of tile coding
commonly include efficient hashing.
Exercise 9.4 Suppose we believe that one of two state dimensions is more likely to havean e↵ect on the value function than is the other, that generalization should be primarily
across this dimension rather than along it. What kind of tilings could be used to takeadvantage of this prior knowledge?
⇤

9.5.5

Radial Basis Functions

Radial basis functions (RBFs) are the natural generalization of coarse coding to continuousvalued features. Rather than each feature being either 0 or 1, it can be anything in theinterval [0, 1], reflecting various degrees to which the feature is present. A typical RBF
feature, xi , has a Gaussian (bell-shaped) response xi (s) dependent only on the distance
between the state, s, and the feature’s prototypical or center state, ci , and relative to thefeature’s width, i :✓◆||s ci ||2.
xi (s) = exp.
2 i2
The norm or distance metric of course can be chosen in whatever way seems mostappropriate to the states and task at hand. The figure below shows a one-dimensional
example with a Euclidean distance metric.

!i

ci-1

ci

ci+1

Figure 9.13: One-dimensional radial basis functions.

222

Chapter 9: On-policy Prediction with Approximation

The primary advantage of RBFs over binary features is that they produce approximate
functions that vary smoothly and are di↵erentiable. Although this is appealing, in mostcases it has no practical significance. Nevertheless, extensive studies have been made ofgraded response functions such as RBFs in the context of tile coding (An, 1991; Miller etal., 1991; An et al., 1991; Lane, Handelman and Gelfand, 1992). All of these methods
require substantial additional computational complexity (over tile coding) and often
reduce performance when there are more than two state dimensions. In high dimensions
the edges of tiles are much more important, and it has proven difficult to obtain wellcontrolled graded tile activations near the edges.
An RBF network is a linear function approximator using RBFs for its features. Learning
is defined by equations (9.7) and (9.8), exactly as in other linear function approximators.
In addition, some learning methods for RBF networks change the centers and widths of
the features as well, bringing them into the realm of nonlinear function approximators.
Nonlinear methods may be able to fit target functions much more precisely. The downside
to RBF networks, and to nonlinear RBF networks especially, is greater computational
complexity and, often, more manual tuning before learning is robust and efficient.

9.6

Selecting Step-Size Parameters Manually

Most SGD methods require the designer to select an appropriate step-size parameter ↵.
Ideally this selection would be automated, and in some cases it has been, but for mostcases it is still common practice to set it manually. To do this, and to better understandthe algorithms, it is useful to develop some intuitive sense of the role of the step-size
parameter. Can we say in general how it should be set?
Theoretical considerations are unfortunately of little help. The theory of stochastic
approximation gives us conditions (2.7) on a slowly decreasing step-size sequence that are
sufficient to guarantee convergence, but these tend to result in learning that is too slow.
The classical choice ↵t = 1/t, which produces sample averages in tabular MC methods, is
not appropriate for TD methods, for nonstationary problems, or for any method using
function approximation. For linear methods, there are recursive least-squares methods
that set an optimal matrix step size, and these methods can be extended to temporaldi↵erence learning as in the LSTD method described in Section 9.8, but these requireO(d2 ) step-size parameters, or d times more parameters than we are learning. For thisreason we rule them out for use on large problems where function approximation is mostneeded.
To get some intuitive feel for how to set the step-size parameter manually, it is best
to go back momentarily to the tabular case. There we can understand that a step sizeof ↵ = 1 will result in a complete elimination of the sample error after one target (see(2.4) with a step size of one). As discussed on page 201, we usually want to learn slower
1
than this. In the tabular case, a step size of ↵ = 10
would take about 10 experiences toconverge approximately to their mean target, and if we wanted to learn in 100 experiences1
we would use ↵ = 100. In general, if ↵ = ⌧1 , then the tabular estimate for a state willapproach the mean of its targets, with the most recent targets having the greatest e↵ect,
after about ⌧ experiences with the state.

9.7. Nonlinear Function Approximation: Artificial Neural Networks

223

With general function approximation there is not such a clear notion of number ofexperiences with a state, as each state may be similar to and dissimilar from all the others
to various degrees. However, there is a similar rule that gives similar behavior in the caseof linear function approximation. Suppose you wanted to learn in about ⌧ experienceswith substantially the same feature vector. A good rule of thumb for setting the step-size
parameter of linear SGD methods is then⇥⇤.
↵ = ⌧ E x> x1,

(9.19)

where x is a random feature vector chosen from the same distribution as input vectors
will be in the SGD. This method works best if the feature vectors do not vary greatly inlength; ideally x> x is a constant.
Exercise 9.5 Suppose you are using tile coding to transform a seven-dimensional continuous
state space into binary feature vectors to estimate a state value function v̂(s,w) ⇡ v⇡ (s).
You believe that the dimensions do not interact strongly, so you decide to use eight tilingsof each dimension separately (stripe tilings), for 7 ⇥ 8 = 56 tilings. In addition, in casethere are some pairwise interactions between the dimensions, you also take all 72 = 21
pairs of dimensions and tile each pair conjunctively with rectangular tiles. You maketwo tilings for each pair of dimensions, making a grand total of 21 ⇥ 2 + 56 = 98 tilings.
Given these feature vectors, you suspect that you still have to average out some noise,so you decide that you want learning to be gradual, taking about 10 presentations withthe same feature vector before learning nears its asymptote. What step-size parameter ↵should you use? Why?
⇤
⇥ > ⇤
>
Exercise 9.6 If ⌧ = 1 and x(St ) x(St ) = E x x , prove that (9.19) together with (9.7)
and linear function approximation results in the error being reduced to zero in one update.

9.7

Nonlinear Function Approximation:
Artificial Neural Networks

Artificial neural networks (ANNs) are widely used for nonlinear function approximation.
An ANN is a network of interconnected units that have some of the properties of neurons,
the main components of nervous systems. ANNs have a long history, with the latest
advances in training deeply-layered ANNs (deep learning) being responsible for someof the most impressive abilities of machine learning systems, including reinforcement
learning systems. In Chapter 16 we describe several impressive examples of reinforcement
learning systems that use ANN function approximation.
Figure 9.14 shows a generic feedforward ANN, meaning that there are no loops in the
network, that is, there are no paths within the network by which a unit’s output can
influence its input. The network in the figure has an output layer consisting of two output
units, an input layer with four input units, and two “hidden layers”: layers that are neither
input nor output layers. A real-valued weight is associated with each link. A weightroughly corresponds to the efficacy of a synaptic connection in a real neural network (seeSection 15.1). If an ANN has at least one loop in its connections, it is a recurrent rather

224

Chapter 9: On-policy Prediction with Approximation

Figure 9.14: A generic feedforward ANN with four input units, two output units, and two
hidden layers.

than a feedforward ANN. Although both feedforward and recurrent ANNs have beenused in reinforcement learning, here we look only at the simpler feedforward case.
The units (the circles in Figure 9.14) are typically semi-linear units, meaning that they
compute a weighted sum of their input signals and then apply to the result a nonlinear
function, called the activation function, to produce the unit’s output, or activation.
Di↵erent activation functions are used, but they are typically S-shaped, or sigmoid,functions such as the logistic function f (x) = 1/(1 + e x ), though sometimes the rectifier
nonlinearity f (x) = max(0, x) is used. A step function like f (x) = 1 if x✓, and 0
otherwise, results in a binary unit with threshold ✓. The units in a network’s input layer
are somewhat di↵erent in having their activations set to externally-supplied values thatare the inputs to the function the network is approximating.
The activation of each output unit of a feedforward ANN is a nonlinear function of theactivation patterns over the network’s input units. The functions are parameterized by
the network’s connection weights. An ANN with no hidden layers can represent only avery small fraction of the possible input-output functions. However an ANN with a single
hidden layer containing a large enough finite number of sigmoid units can approximate
any continuous function on a compact region of the network’s input space to any degreeof accuracy (Cybenko, 1989). This is also true for other nonlinear activation functions
that satisfy mild conditions, but nonlinearity is essential: if all the units in a multi-layer
feedforward ANN have linear activation functions, the entire network is equivalent to a
network with no hidden layers (because linear functions of linear functions are themselveslinear).
Despite this “universal approximation” property of one-hidden-layer ANNs, bothexperience and theory show that approximating the complex functions needed for many
artificial intelligence tasks is made easier—indeed may require—abstractions that are
hierarchical compositions of many layers of lower-level abstractions, that is, abstractions

9.7. Nonlinear Function Approximation: Artificial Neural Networks

225

produced by deep architectures such as ANNs with many hidden layers. (See Bengio,
2009, for a thorough review.) The successive layers of a deep ANN compute increasingly
abstract representations of the network’s “raw” input, with each unit providing a feature
contributing to a hierarchical representation of the overall input-output function of the
network.
Training the hidden layers of an ANN is therefore a way to automatically create
features appropriate for a given problem so that hierarchical representations can beproduced without relying exclusively on hand-crafted features. This has been an enduring
challenge for artificial intelligence and explains why learning algorithms for ANNs withhidden layers have received so much attention over the years. ANNs typically learn by astochastic gradient method (Section 9.3). Each weight is adjusted in a direction aimed at
improving the network’s overall performance as measured by an objective function tobe either minimized or maximized. In the most common supervised learning case, theobjective function is the expected error, or loss, over a set of labeled training examples. In
reinforcement learning, ANNs can use TD errors to learn value functions, or they can aim
to maximize expected reward as in a gradient bandit (Section 2.8) or a policy-gradientalgorithm (Chapter 13). In all of these cases it is necessary to estimate how a change
in each connection weight would influence the network’s overall performance, in otherwords, to estimate the partial derivative of an objective function with respect to eachweight, given the current values of all the network’s weights. The gradient is the vectorof these partial derivatives.
The most successful way to do this for ANNs with hidden layers (provided the units
have di↵erentiable activation functions) is the backpropagation algorithm, which consistsof alternating forward and backward passes through the network. Each forward passcomputes the activation of each unit given the current activations of the network’s input
units. After each forward pass, a backward pass efficiently computes a partial derivativefor each weight. (As in other stochastic gradient learning algorithms, the vector of thesepartial derivatives is an estimate of the true gradient.) In Section 15.10 we discuss
methods for training ANNs with hidden layers that use reinforcement learning principles
instead of backpropagation. These methods are less efficient than the backpropagation
algorithm, but they may be closer to how real neural networks learn.
The backpropagation algorithm can produce good results for shallow networks having
1 or 2 hidden layers, but it may not work well for deeper ANNs. In fact, training a
network with k + 1 hidden layers can actually result in poorer performance than training
a network with k hidden layers, even though the deeper network can represent all thefunctions that the shallower network can (Bengio, 2009). Explaining results like theseis not easy, but several factors are important. First, the large number of weights ina typical deep ANN makes it difficult to avoid the problem of overfitting, that is, theproblem of failing to generalize correctly to cases on which the network has not been
trained. Second, backpropagation does not work well for deep ANNs because the partial
derivatives computed by its backward passes either decay rapidly toward the input sideof the network, making learning by deep layers extremely slow, or the partial derivatives
grow rapidly toward the input side of the network, making learning unstable. Methods
for dealing with these problems are largely responsible for many impressive recent results

226

Chapter 9: On-policy Prediction with Approximationachieved by systems that use deep ANNs.
Overfitting is a problem for any function approximation method that adjusts functions
with many degrees of freedom on the basis of limited training data. It is less of aproblem for online reinforcement learning that does not rely on limited training sets, but
generalizing e↵ectively is still an important issue. Overfitting is a problem for ANNs ingeneral, but especially so for deep ANNs because they tend to have very large numbers
of weights. Many methods have been developed for reducing overfitting. These include
stopping training when performance begins to decrease on validation data di↵erent
from the training data (cross validation), modifying the objective function to discourage
complexity of the approximation (regularization), and introducing dependencies among
the weights to reduce the number of degrees of freedom (e.g., weight sharing).
A particularly e↵ective method for reducing overfitting by deep ANNs is the dropout
method introduced by Srivastava, Hinton, Krizhevsky, Sutskever, and Salakhutdinov(2014). During training, units are randomly removed from the network (dropped out)
along with their connections. This can be thought of as training a large number of“thinned” networks. Combining the results of these thinned networks at test time is a way
to improve generalization performance. The dropout method efficiently approximates thiscombination by multiplying each outgoing weight of a unit by the probability that that
unit was retained during training. Srivastava et al. found that this method significantly
improves generalization performance. It encourages individual hidden units to learn
features that work well with random collections of other features. This increases theversatility of the features formed by the hidden units so that the network does not overly
specialize to rarely-occurring cases.
Hinton, Osindero, and Teh (2006) took a major step toward solving the problem oftraining the deep layers of a deep ANN in their work with deep belief networks, layered
networks closely related to the deep ANNs discussed here. In their method, the deepest
layers are trained one at a time using an unsupervised learning algorithm. Without
relying on the overall objective function, unsupervised learning can extract features that
capture statistical regularities of the input stream. The deepest layer is trained first, thenwith input provided by this trained layer, the next deepest layer is trained, and so on,
until the weights in all, or many, of the network’s layers are set to values that now act asinitial values for supervised learning. The network is then fine-tuned by backpropagation
with respect to the overall objective function. Studies show that this approach generally
works much better than backpropagation with weights initialized with random values.
The better performance of networks trained with weights initialized this way could be
due to many factors, but one idea is that this method places the network in a region ofweight space from which a gradient-based algorithm can make good progress.
Batch normalization (Io↵e and Szegedy, 2015) is another technique that makes it easierto train deep ANNs. It has long been known that ANN learning is easier if the network
input is normalized, for example, by adjusting each input variable to have zero mean andunit variance. Batch normalization for training deep ANNs normalizes the output of deep
layers before they feed into the following layer. Io↵e and Szegedy (2015) used statistics
from subsets, or “mini-batches,” of training examples to normalize these between-layer
signals to improve the learning rate of deep ANNs.

9.7. Nonlinear Function Approximation: Artificial Neural Networks

227

Another technique useful for training deep ANNs is deep residual learning (He, Zhang,Ren, and Sun, 2016). Sometimes it is easier to learn how a function di↵ers from theidentity function than to learn the function itself. Then adding this di↵erence, or residualfunction, to the input produces the desired function. In deep ANNs, a block of layers
can be made to learn a residual function simply by adding shortcut, or skip, connections
around the block. These connections add the input to the block to its output, andno additional weights are needed. He et al. (2016) evaluated this method using deep
convolutional networks with skip connections around every pair of adjacent layers, finding
substantial improvement over networks without the skip connections on benchmark image
classification tasks. Both batch normalization and deep residual learning were used in
the reinforcement learning application to the game of Go that we describe in Chapter 16.
A type of deep ANN that has proven to be very successful in applications, including
impressive reinforcement learning applications (Chapter 16), is the deep convolutional
network. This type of network is specialized for processing high-dimensional data arrangedin spatial arrays, such as images. It was inspired by how early visual processing works inthe brain (LeCun, Bottou, Bengio and Ha↵ner, 1998). Because of its special architecture,a deep convolutional network can be trained by backpropagation without resorting tomethods like those described above to train the deep layers.
Figure 9.15 illustrates the architecture of a deep convolutional network. This instance,from LeCun et al. (1998), was designed to recognize hand-written characters. It consistsof alternating convolutional and subsampling layers, followed by several fully connected
final layers. Each convolutional layer produces a number of feature maps. A feature
map is a pattern of activity over an array of units, where each unit performs the same
operation on data in its receptive field, which is the part of the data it “sees” from thepreceding layer (or from the external input in the case of the first convolutional layer).
The units of a feature map are identical to one another except that their receptive fields,
which are all the same size and shape, are shifted to di↵erent locations on the arraysof incoming data. Units in the same feature map share the same weights. This meansthat a feature map detects the same feature no matter where it is located in the input

Figure 9.15: Deep Convolutional Network. Republished with permission of Proceedings of theIEEE, from Gradient-based learning applied to document recognition, LeCun, Bottou, Bengio,and Ha↵ner, volume 86, 1998; permission conveyed through Copyright Clearance Center, Inc.

228

Chapter 9: On-policy Prediction with Approximationarray. In the network in Figure 9.15, for example, the first convolutional layer produces
6 feature maps, each consisting of 28 ⇥ 28 units. Each unit in each feature map has a
5 ⇥ 5 receptive field, and these receptive fields overlap (in this case by four columns andfour rows). Consequently, each of the 6 feature maps is specified by just 25 adjustableweights.
The subsampling layers of a deep convolutional network reduce the spatial resolution of
the feature maps. Each feature map in a subsampling layer consists of units that averageover a receptive field of units in the feature maps of the preceding convolutional layer.
For example, each unit in each of the 6 feature maps in the first subsampling layer of the
network of Figure 9.15 averages over a 2 ⇥ 2 non-overlapping receptive field over one of
the feature maps produced by the first convolutional layer, resulting in six 14 ⇥ 14 featuremaps. Subsampling layers reduce the network’s sensitivity to the spatial locations of thefeatures detected, that is, they help make the network’s responses spatially invariant.
This is useful because a feature detected at one place in an image is likely to be useful atother places as well.
Advances in the design and training of ANNs—of which we have only mentioned a
few—all contribute to reinforcement learning. Although current reinforcement learningtheory is mostly limited to methods using tabular or linear function approximation
methods, the impressive performances of notable reinforcement learning applications owe
much of their success to nonlinear function approximation by multi-layer ANNs. We
discuss several of these applications in Chapter 16.

9.8

Least-Squares TD

All the methods we have discussed so far in this chapter have required computation pertime step proportional to the number of parameters. With more computation, however,
one can do better. In this section we present a method for linear function approximationthat is arguably the best that can be done for this case.
As we established in Section 9.4 TD(0) with linear function approximation converges
asymptotically (for appropriately decreasing step sizes) to the TD fixed point:
wTD = A 1 b,
where
. ⇥
A = E xt (xtxt+1 )>⇤

and.
b = E[Rt+1 xt ] .

Why, one might ask, must we compute this solution iteratively? This is wasteful of data!
Could one not do better by computing estimates of A and b, and then directly computing
the TD fixed point? The Least-Squares TD algorithm, commonly known as LSTD, does
exactly this. It forms the natural estimates
t 1

. X
bt =
A
xk (xkk=0

t 1

xk+1 )> + "I

and

. X
bt =
b
Rk+1 xk ,k=0(9.20)

9.8. Least-Squares TD229

b t is always
where I is the identity matrix, and "I, for some small " > 0, ensures that A
invertible. It might seem that these estimates should both be divided by t, and indeed
they should; as defined here, these are really estimates of t times A and t times b.
However, the extra t factors cancel out when LSTD uses these estimates to estimate theTD fixed point as. b 1bwt = A(9.21)
t bt .
This algorithm is the most data efficient form of linear TD(0), but it is also more
expensive computationally. Recall that semi-gradient TD(0) requires memory and perstep computation that is only O(d).
How complex is LSTD? As it is written above the complexity seems to increase witht, but the two approximations in (9.20) could be implemented incrementally using thetechniques we have covered earlier (e.g., in Chapter 2) so that they can be done inb t would involve an outer product (aconstant time per step. Even so, the update for A
column vector times a row vector) and thus would be a matrix update; its computationalb t matrix
complexity would be O(d2 ), and of course the memory required to hold the A2
would be O(d ).
A potentially greater problem is that our final computation (9.21) uses the inverseb t , and the computational complexity of a general inverse computation is O(d3 ).
of A
Fortunately, an inverse of a matrix of our special form—a sum of outer products—canalso be updated incrementally with only O(d2 ) computations, as⇣⌘ 1
b 1= Ab t 1 + xt 1 (xt 1A
xt )>(from (9.20))
t
b 1
=A
t 1

b 1 xt 1 (xt 1
A
t 1
1 + (xt 1

b 1xt )> At 1,
1>bxt ) A xt 1

(9.22)t 1.
b0 =for t > 0, with A"I. Although the identity (9.22), known as the Sherman-Morrison
formula, is superficially complicated, it involves only vector-matrix and vector-vectorb 1,
multiplications and thus is only O(d2 ). Thus we can store the inverse matrix Atmaintain it with (9.22), and then use it in (9.21), all with only O(d2 ) memory andper-step computation. The complete algorithm is given in the box on the next page.
Of course, O(d2 ) is still significantly more expensive than the O(d) of semi-gradient
TD. Whether the greater data efficiency of LSTD is worth this computational expensedepends on how large d is, how important it is to learn quickly, and the expense of otherparts of the system. The fact that LSTD requires no step-size parameter is sometimes
also touted, but the advantage of this is probably overstated. LSTD does not require a
step size, but it does requires "; if " is chosen too small the sequence of inverses can vary
wildly, and if " is chosen too large then learning is slowed. In addition, LSTD’s lack of astep-size parameter means that it never forgets. This is sometimes desirable, but it isproblematic if the target policy ⇡ changes as it does in reinforcement learning and GPI.
In control applications, LSTD typically has to be combined with some other mechanism
to induce forgetting, mooting any initial advantage of not requiring a step-size parameter.

230

Chapter 9: On-policy Prediction with Approximation

LSTD for estimating v̂ = w> x(·) ⇡ v⇡ (O(d2 ) version)
Input: feature representation x : S+ ! Rd such that x(terminal) = 0
Algorithm parameter: small " > 0
d1
A
" 1I
A d ⇥ d matrixbb0
A d-dimensional vector
Loop for each episode:
Initialize S; xx(S)Loop for each step of episode:
Choose and take action A ⇠ ⇡(·|S), observe R, S 0 ; x0x(S 0 )
>
d1 (x
v
A
x0 )
d
d
d
A 1
A 1
A 1 x v> / 1 + v> xbb + Rx
b
b
d1 b
b
w
A
0
S
S;x
x0
0
until S is terminal

9.9

Memory-based Function Approximation

So far we have discussed the parametric approach to approximating value functions. In
this approach, a learning algorithm adjusts the parameters of a functional form intended
to approximate the value function over a problem’s entire state space. Each update,s 7! g, is a training example used by the learning algorithm to change the parameterswith the aim of reducing the approximation error. After the update, the training example
can be discarded (although it might be saved to be used again). When an approximate
value of a state (which we will call the query state) is needed, the function is simply
evaluated at that state using the latest parameters produced by the learning algorithm.
Memory-based function approximation methods are very di↵erent. They simply savetraining examples in memory as they arrive (or at least save a subset of the examples)
without updating any parameters. Then, whenever a query state’s value estimate isneeded, a set of examples is retrieved from memory and used to compute a value estimate
for the query state. This approach is sometimes called lazy learning because processing
training examples is postponed until the system is queried to provide an output.
Memory-based function approximation methods are prime examples of nonparametricmethods. Unlike parametric methods, the approximating function’s form is not limited
to a fixed parameterized class of functions, such as linear functions or polynomials, but isinstead determined by the training examples themselves, together with some means for
combining them to output estimated values for query states. As more training examples
accumulate in memory, one expects nonparametric methods to produce increasingly
accurate approximations of any target function.

9.9. Memory-based Function Approximation

231

There are many di↵erent memory-based methods depending on how the stored training
examples are selected and how they are used to respond to a query. Here, we focus on
local-learning methods that approximate a value function only locally in the neighborhoodof the current query state. These methods retrieve a set of training examples from memory
whose states are judged to be the most relevant to the query state, where relevance
usually depends on the distance between states: the closer a training example’s state isto the query state, the more relevant it is considered to be, where distance can be definedin many di↵erent ways. After the query state is given a value, the local approximation is
discarded.
The simplest example of the memory-based approach is the nearest neighbor method,which simply finds the example in memory whose state is closest to the query state andreturns that example’s value as the approximate value of the query state. In other words,
if the query state is s, and s0 7! g is the example in memory in which s0 is the closest
state to s, then g is returned as the approximate value of s. Slightly more complicatedare weighted average methods that retrieve a set of nearest neighbor examples and returna weighted average of their target values, where the weights generally decrease withincreasing distance between their states and the query state. Locally weighted regression issimilar, but it fits a surface to the values of a set of nearest states by means of a parametricapproximation method that minimizes a weighted error measure like (9.1), where theweights depend on distances from the query state. The value returned is the evaluation of
the locally-fitted surface at the query state, after which the local approximation surface
is discarded.
Being nonparametric, memory-based methods have the advantage over parametricmethods of not limiting approximations to pre-specified functional forms. This allows
accuracy to improve as more data accumulates. Memory-based local approximation
methods have other properties that make them well suited for reinforcement learning.
Because trajectory sampling is of such importance in reinforcement learning, as discussed
in Section 8.6, memory-based local methods can focus function approximation on local
neighborhoods of states (or state–action pairs) visited in real or simulated trajectories.
There may be no need for global approximation because many areas of the state space will
never (or almost never) be reached. In addition, memory-based methods allow an agent’s
experience to have a relatively immediate a↵ect on value estimates in the neighborhoodof the current state, in contrast with a parametric method’s need to incrementally adjust
parameters of a global approximation.
Avoiding global approximation is also a way to address the curse of dimensionality.
For example, for a state space with k dimensions, a tabular method storing a global
approximation requires memory exponential in k. On the other hand, in storing examples
for a memory-based method, each example requires memory proportional to k, and thememory required to store, say, n examples is linear in n. Nothing is exponential in k orn. Of course, the critical remaining issue is whether a memory-based method can answer
queries quickly enough to be useful to an agent. A related concern is how speed degradesas the size of the memory grows. Finding nearest neighbors in a large database can take
too long to be practical in many applications.

232

Chapter 9: On-policy Prediction with Approximation

Proponents of memory-based methods have developed ways to accelerate the nearest
neighbor search. Using parallel computers or special purpose hardware is one approach;another is the use of special multi-dimensional data structures to store the training data.
One data structure studied for this application is the k-d tree (short for k-dimensionaltree), which recursively splits a k-dimensional space into regions arranged as nodes of abinary tree. Depending on the amount of data and how it is distributed over the state
space, nearest-neighbor search using k-d trees can quickly eliminate large regions of thespace in the search for neighbors, making the searches feasible in some problems where
naive searches would take too long.
Locally weighted regression additionally requires fast ways to do the local regression
computations which have to be repeated to answer each query. Researchers have developed
many ways to address these problems, including methods for forgetting entries in order tokeep the size of the database within bounds. The Bibliographic and Historical Comments
section at the end of this chapter points to some of the relevant literature, including a
selection of papers describing applications of memory-based learning to reinforcement
learning.

9.10

Kernel-based Function Approximation

Memory-based methods such as the weighted average and locally weighted regression
methods described above depend on assigning weights to examples s0 7! g in the databasedepending on the distance between s0 and a query states s. The function that assigns
these weights is called a kernel function, or simply a kernel. In the weighted average andlocally weighted regressions methods, for example, a kernel function k : R ! R assigns
weights to distances between states. More generally, weights do not have to depend on
distances; they can depend on some other measure of similarity between states. In thiscase, k : S ⇥ S ! R, so that k(s, s0 ) is the weight given to data about s0 in its influenceon answering queries about s.
Viewed slightly di↵erently, k(s, s0 ) is a measure of the strength of generalization from0s to s. Kernel functions numerically express how relevant knowledge about any stateis to any other state. As an example, the strengths of generalization for tile codingshown in Figure 9.11 correspond to di↵erent kernel functions resulting from uniform andasymmetrical tile o↵sets. Although tile coding does not explicitly use a kernel functionin its operation, it generalizes according to one. In fact, as we discuss more below, thestrength of generalization resulting from linear parametric function approximation canalways be described by a kernel function.
Kernel regression is the memory-based method that computes a kernel weighted averageof the targets of all examples stored in memory, assigning the result to the query state.
If D is the set of stored examples, and g(s0 ) denotes the target for state s0 in a stored
example, then kernel regression approximates the target function, in this case a valuefunction depending on D, asXv̂(s,D) =k(s, s0 )g(s0 ).
(9.23)
s0 2D

9.10. Kernel-based Function Approximation

233

The weighted average method described above is a special case in which k(s, s0 ) is non-zero
only when s and s0 are close to one another so that the sum need not be computed over
all of D.
A common kernel is the Gaussian radial basis function (RBF) used in RBF function
approximation as described in Section 9.5.5. In the method described there, RBFs arefeatures whose centers and widths are either fixed from the start, with centers presumably
concentrated in areas where many examples are expected to fall, or are adjusted in someway during learning. Barring methods that adjust centers and widths, this is a linear
parametric method whose parameters are the weights of each RBF, which are typicallylearned by stochastic gradient, or semi-gradient, descent. The form of the approximation
is a linear combination of the pre-determined RBFs. Kernel regression with an RBF
kernel di↵ers from this in two ways. First, it is memory-based: the RBFs are centered on
the states of the stored examples. Second, it is nonparametric: there are no parameters
to learn; the response to a query is given by (9.23).
Of course, many issues have to be addressed for practical implementation of kernel
regression, issues that are beyond the scope of our brief discussion. However, it turns outthat any linear parametric regression method like those we described in Section 9.4, withstates represented by feature vectors x(s) = (x1 (s), x2 (s), . . . , xd (s))> , can be recast as
kernel regression where k(s, s0 ) is the inner product of the feature vector representationsof s and s0 ; that isk(s, s0 ) = x(s)> x(s0 ).

(9.24)

Kernel regression with this kernel function produces the same approximation that a linear
parametric method would if it used these feature vectors and learned with the same
training data.
We skip the mathematical justification for this, which can be found in any modern
machine learning text, such as Bishop (2006), and simply point out an important
implication. Instead of constructing features for linear parametric function approximators,
one can instead construct kernel functions directly without referring at all to feature
vectors. Not all kernel functions can be expressed as inner products of feature vectors
as in (9.24), but a kernel function that can be expressed like this can o↵er significant
advantages over the equivalent parametric method. For many sets of feature vectors,(9.24) has a compact functional form that can be evaluated without any computation
taking place in the d-dimensional feature space. In these cases, kernel regression is muchless complex than directly using a linear parametric method with states represented bythese feature vectors. This is the so-called “kernel trick” that allows e↵ectively working
in the high-dimension of an expansive feature space while actually working only with theset of stored training examples. The kernel trick is the basis of many machine learning
methods, and researchers have shown how it can sometimes benefit reinforcement learning.

234

Chapter 9: On-policy Prediction with Approximation

9.11

Looking Deeper at On-policy Learning:
Interest and Emphasis

The algorithms we have considered so far in this chapter have treated all the statesencountered equally, as if they were all equally important. In some cases, however, we
are more interested in some states than others. In discounted episodic problems, forexample, we may be more interested in accurately valuing early states in the episodethan in later states where discounting may have made the rewards much less importantto the value of the start state. Or, if an action-value function is being learned, it may beless important to accurately value poor actions whose value is much less than the greedy
action. Function approximation resources are always limited, and if they were used in amore targeted way, then performance could be improved.
One reason we have treated all states encountered equally is that then we are updating
according to the on-policy distribution, for which stronger theoretical results are available
for semi-gradient methods. Recall that the on-policy distribution was defined as the
distribution of states encountered in an MDP while following the target policy. Now wewill generalize this concept significantly. Rather than having one on-policy distribution
for the MDP, we will have many. All of them will have in common that they are a
distribution of states encountered in trajectories while following the target policy, butthey will vary in how the trajectories are, in a sense, initiated.
We now introduce some new concepts. First we introduce a non-negative scalar measure,a random variable It called interest, indicating the degree to which we are interested inaccurately valuing the state (or state–action pair) at time t. If we don’t care at all about
the state, then the interest should be zero; if we fully care, it might be one, though it isformally allowed to take any non-negative value. The interest can be set in any causal
way; for example, it may depend on the trajectory up to time t or the learned parameters
at time t. The distribution µ in the VE (9.1) is then defined as the distribution ofstates encountered while following the target policy, weighted by the interest. Second, we
introduce another non-negative scalar random variable, the emphasis Mt . This scalarmultiplies the learning update and thus emphasizes or de-emphasizes the learning done
at time t. The general n-step learning rule, replacing (9.15), is.
wt+n = wt+n 1 + ↵Mt [Gt:t+nv̂(St ,wt+n 1 )] rv̂(St ,wt+n 1 ),0  t < T, (9.25)

with the n-step return given by (9.16) and the emphasis determined recursively from theinterest by:
Mt = I t +n

Mt n ,

0  t < T,

(9.26).
with Mt = 0, for all t < 0. These equations are taken to include the Monte Carlo case,for which Gt:t+n = Gt , all the updates are made at end of the episode, n = T t, and
Mt = I t .

9.11. Looking Deeper at On-policy Learning: Interest and Emphasis

235

Example 9.4 illustrates how interest and emphasis can result in more accurate value
estimates.
Example 9.4: Interest and Emphasis
To see the potential benefits of using interest and emphasis, consider the four-state
Markov reward process shown below:

i=0

i=1

w1
v⇡ = 4+1

w1v⇡ = 3i=0+1w2v⇡ = 2

i=0

+1

w2

+1

v⇡ = 1

Episodes start in the leftmost state, then transition one state to the right, with areward of +1, on each step until the terminal state is reached. The true value ofthe first state is thus 4, of the second state 3, and so on as shown below each state.
These are the true values; the estimated values can only approximate these because
they are constrained by the parameterization. There are two components to theparameter vector w = (w1 , w2 )> , and the parameterization is as written insideeach state. The estimated values of the first two states are given by w1 alone andthus must be the same even though their true values are di↵erent. Similarly, theestimated values of the third and fourth states are given by w2 alone and must be
the same even though their true values are di↵erent. Suppose that we are interested
in accurately valuing only the leftmost state; we assign it an interest of 1 while all
the other states are assigned an interest of 0, as indicated above the states.
First consider applying gradient Monte Carlo algorithms to this problem. The
algorithms presented earlier in this chapter that do not take into account interest
and emphasis (in (9.7) and the box on page 202) will converge (for decreasing step
sizes) to the parameter vector w1 = (3.5, 1.5), which gives the first state—the only
one we are interested in—a value of 3.5 (i.e., intermediate between the true valuesof the first and second states). The methods presented in this section that do useinterest and emphasis, on the other hand, will learn the value of the first state
exactly correctly; w1 will converge to 4 while w2 will never be updated because theemphasis is zero in all states save the leftmost.
Now consider applying two-step semi-gradient TD methods. The methods fromearlier in this chapter without interest and emphasis (in (9.15) and (9.16) andthe box on page 209) will again converge to w1 = (3.5, 1.5), while the methods
with interest and emphasis converge to w1 = (4, 2). The latter produces theexactly correct values for the first state and for the third state (which the first statebootstraps from) while never making any updates corresponding to the second or
fourth states.

236

9.12

Chapter 9: On-policy Prediction with Approximation

Summary

Reinforcement learning systems must be capable of generalization if they are to be
applicable to artificial intelligence or to large engineering applications. To achieve this,any of a broad range of existing methods for supervised-learning function approximation
can be used simply by treating each update as a training example.
Perhaps the most suitable supervised learning methods are those using parameterizedfunction approximation, in which the policy is parameterized by a weight vector w.
Although the weight vector has many components, the state space is much larger still,
and we must settle for an approximate solution. We defined the mean square value error,
VE(w), as a measure of the error in the values v⇡w (s) for a weight vector w under the
on-policy distribution, µ. The VE gives us a clear way to rank di↵erent value-function
approximations in the on-policy case.
To find a good weight vector, the most popular methods are variations of stochastic
gradient descent (SGD). In this chapter we have focused on the on-policy case with a fixed
policy, also known as policy evaluation or prediction; a natural learning algorithm for thiscase is n-step semi-gradient TD, which includes gradient Monte Carlo and semi-gradientTD(0) algorithms as the special cases when n = 1 and n = 1 respectively. Semi-gradient
TD methods are not true gradient methods. In such bootstrapping methods (includingDP), the weight vector appears in the update target, yet this is not taken into account incomputing the gradient—thus they are semi -gradient methods. As such, they cannot
rely on classical SGD results.
Nevertheless, good results can be obtained for semi-gradient methods in the special caseof linear function approximation, in which the value estimates are sums of features times
corresponding weights. The linear case is the most well understood theoretically andworks well in practice when provided with appropriate features. Choosing the features
is one of the most important ways of adding prior domain knowledge to reinforcement
learning systems. They can be chosen as polynomials, but this case generalizes poorly inthe online learning setting typically considered in reinforcement learning. Better is tochoose features according the Fourier basis, or according to some form of coarse codingwith sparse overlapping receptive fields. Tile coding is a form of coarse coding thatis particularly computationally efficient and flexible. Radial basis functions are usefulfor one- or two-dimensional tasks in which a smoothly varying response is important.
LSTD is the most data-efficient linear TD prediction method, but requires computation
proportional to the square of the number of weights, whereas all the other methods are ofcomplexity linear in the number of weights. Nonlinear methods include artificial neuralnetworks trained by backpropagation and variations of SGD; these methods have become
very popular in recent years under the name deep reinforcement learning.
Linear semi-gradient n-step TD is guaranteed to converge under standard conditions,for all n, to a VE that is within a bound of the optimal error (achieved asymptoticallyby Monte Carlo methods). This bound is always tighter for higher n and approaches
zero as n ! 1. However, in practice very high n results in very slow learning, and somedegree of bootstrapping (n < 1) is usually preferable, just as we saw in comparisons oftabular n-step methods in Chapter 7 and in comparisons of tabular TD and Monte Carlo
methods in Chapter 6.

9.12. Summary

237

Exercise 9.7 One of the simplest artificial neural networks consists of a single semi-linear
unit with a logistic nonlinearity. The need to handle approximate value functions of thisform is common in games that end with either a win or a loss, in which case the value ofa state can be interpreted as the probability of winning. Derive the learning algorithm
for this case, from (9.7), such that no gradient notation appears.
⇤

Exercise 9.8 Arguably, the squared error used to derive (9.7) is inappropriate for thecase treated in the preceding exercise, and the right error measure is the cross-entropy
loss (which you can find on Wikipedia). Repeat the derivation in Section 9.3, using thecross-entropy loss instead of the squared error in (9.4), all the way to an explicit form
with no gradient or logarithm notation in it. Is your final form more complex, or simpler,than that you obtained in the preceding exercise?

Bibliographical and Historical Remarks
Generalization and function approximation have always been an integral part of reinforcement learning. Bertsekas and Tsitsiklis (1996), Bertsekas (2012), and Sugiyama etal. (2013) present the state of the art in function approximation in reinforcement learning.
Some of the early work with function approximation in reinforcement learning is discussed
at the end of this section.
9.3

Gradient-descent methods for minimizing mean square error in supervised learning
are well known. Widrow and Ho↵ (1960) introduced the least-mean-square (LMS)
algorithm, which is the prototypical incremental gradient-descent algorithm.
Details of this and related algorithms are provided in many texts (e.g., Widrow
and Stearns, 1985; Bishop, 1995; Duda and Hart, 1973).
Semi-gradient TD(0) was first explored by Sutton (1984, 1988), as part of thelinear TD( ) algorithm that we will treat in Chapter 12. The term “semi-gradient”
to describe these bootstrapping methods is new to the second edition of thisbook.
The earliest use of state aggregation in reinforcement learning may have been
Michie and Chambers’s BOXES system (1968). The theory of state aggregationin reinforcement learning has been developed by Singh, Jaakkola, and Jordan(1995) and Tsitsiklis and Van Roy (1996). State aggregation has been used indynamic programming from its earliest days (e.g., Bellman, 1957a).

9.4

Sutton (1988) proved convergence of linear TD(0) in the mean to the minimal
VE solution for the case in which the feature vectors, {x(s) : s 2 S}, are linearly
independent. Convergence with probability 1 was proved by several researchersat about the same time (Peng, 1993; Dayan and Sejnowski, 1994; Tsitsiklis,
1994; Gurvits, Lin, and Hanson, 1994). In addition, Jaakkola, Jordan, and Singh(1994) proved convergence under online updating. All of these results assumed
linearly independent feature vectors, which implies at least as many components
to wt as there are states. Convergence for the more important case of general(dependent) feature vectors was first shown by Dayan (1992). A significant

238

Chapter 9: On-policy Prediction with Approximation

generalization and strengthening of Dayan’s result was proved by Tsitsiklis and
Van Roy (1997). They proved the main result presented in this section, thebound on the asymptotic error of linear bootstrapping methods.
9.5

Our presentation of the range of possibilities for linear function approximation isbased on that by Barto (1990).

9.5.2

Konidaris, Osentoski, and Thomas (2011) introduced the Fourier basis in a
simple form suitable for reinforcement learning problems with multi-dimensional
continuous state spaces and functions that do not have to be periodic.

9.5.3

The term coarse coding is due to Hinton (1984), and our Figure 9.6 is based on
one of his figures. Waltz and Fu (1965) provide an early example of this type offunction approximation in a reinforcement learning system.

9.5.4

Tile coding, including hashing, was introduced by Albus (1971, 1981). He described it in terms of his “cerebellar model articulator controller,” or CMAC, astile coding is sometimes known in the literature. The term “tile coding” was new
to the first edition of this book, though the idea of describing CMAC in theseterms is taken from Watkins (1989). Tile coding has been used in many reinforcement learning systems (e.g., Shewchuk and Dean, 1990; Lin and Kim, 1991;
Miller, Scalera, and Kim, 1994; Sofge and White, 1992; Tham, 1994; Sutton, 1996;
Watkins, 1989) as well as in other types of learning control systems (e.g., Kraft andCampagna, 1990; Kraft, Miller, and Dietz, 1992). This section draws heavily on
the work of Miller and Glanz (1996). General software for tile coding is available inseveral languages (e.g., see http://incompleteideas.net/tiles/tiles3.html).

9.5.5

Function approximation using radial basis functions has received wide attention
ever since being related to ANNs by Broomhead and Lowe (1988). Powell (1987)
reviewed earlier uses of RBFs, and Poggio and Girosi (1989, 1990) extensively
developed and applied this approach.

9.6

Automatic methods for adapting the step-size parameter include RMSprop (Tieleman and Hinton, 2012), Adam (Kingma and Ba, 2015), stochastic meta-descentmethods such as Delta-Bar-Delta (Jacobs, 1988), its incremental generalization (Sutton, 1992b, c; Mahmood et al., 2012), and nonlinear generalizations(Schraudolph, 1999, 2002). Methods explicitly designed for reinforcement learning include AlphaBound (Dabney and Barto, 2012), SID and NOSID (Dabney,2014), TIDBD (Kearney et al., in preparation) and the application of stochastic
meta-descent to policy gradient learning (Schraudolph, Yu, and Aberdeen, 2006).

9.7

The introduction of the threshold logic unit as an abstract model neuron by
McCulloch and Pitts (1943) was the beginning of ANNs. The history of ANNs as
learning methods for classification or regression has passed through several stages:
roughly, the Perceptron (Rosenblatt, 1962) and ADALINE (ADAptive LINear
Element) (Widrow and Ho↵, 1960) stage of learning by single-layer ANNs, the9.12. Summary239

error-backpropagation stage (LeCun, 1985; Rumelhart, Hinton, and Williams,
1986) of learning by multi-layer ANNs, and the current deep-learning stage with
its emphasis on representation learning (e.g., Bengio, Courville, and Vincent,
2012; Goodfellow, Bengio, and Courville, 2016). Examples of the many books on
ANNs are Haykin (1994), Bishop (1995), and Ripley (2007).
ANNs as function approximation for reinforcement learning goes back to the early
work of Farley and Clark (1954), who used reinforcement-like learning to modify
the weights of linear threshold functions representing policies. Widrow, Gupta,
and Maitra (1973) presented a neuron-like linear threshold unit implementing alearning process they called learning with a critic or selective bootstrap adaptation,a reinforcement-learning variant of the ADALINE algorithm. Werbos (1987,
1994) developed an approach to prediction and control that uses ANNs trained byerror backpropation to learn policies and value functions using TD-like algorithms.
Barto, Sutton, and Brouwer (1981) and Barto and Sutton (1981b) extended theidea of an associative memory network (e.g., Kohonen, 1977; Anderson, Silverstein,
Ritz, and Jones, 1977) to reinforcement learning. Barto, Anderson, and Sutton(1982) used a two-layer ANN to learn a nonlinear control policy, and emphasized
the first layer’s role of learning a suitable representation. Hampson (1983, 1989)
was an early proponent of multilayer ANNs for learning value functions. Barto,
Sutton, and Anderson (1983) presented an actor–critic algorithm in the form of an
ANN learning to balance a simulated pole (see Sections 15.7 and 15.8). Barto andAnandan (1985) introduced a stochastic version of Widrow et al.’s (1973) selective
bootstrap algorithm called the associative reward-penalty (AR P ) algorithm.
Barto (1985, 1986) and Barto and Jordan (1987) described multi-layer ANNsconsisting of AR P units trained with a globally-broadcast reinforcement signal
to learn classification rules that are not linearly separable. Barto (1985) discussed
this approach to ANNs and how this type of learning rule is related to others in
the literature at that time. (See Section 15.10 for additional discussion of thisapproach to training multi-layer ANNs.) Anderson (1986, 1987, 1989) evaluatednumerous methods for training multilayer ANNs and showed that an actor–critic
algorithm in which both the actor and critic were implemented by two-layer
ANNs trained by error backpropagation outperformed single-layer ANNs in thepole-balancing and tower of Hanoi tasks. Williams (1988) described several waysthat backpropagation and reinforcement learning can be combined for training
ANNs. Gullapalli (1990) and Williams (1992) devised reinforcement learningalgorithms for neuron-like units having continuous, rather than binary, outputs.
Barto, Sutton, and Watkins (1990) argued that ANNs can play significant roles
for approximating functions required for solving sequential decision problems.
Williams (1992) related REINFORCE learning rules (Section 13.3) to the error
backpropagation method for training multi-layer ANNs. Tesauro’s TD-Gammon(Tesauro 1992, 1994; Section 16.1) influentially demonstrated the learning abilities
of TD( ) algorithm with function approximation by multi-layer ANNs in learning
to play backgammon. The AlphaGo, AlphaGo Zero, and AlphaZero programsof Silver et al. (2016, 2017a, b; Section 16.6) used reinforcement learning with

240

Chapter 9: On-policy Prediction with Approximation

deep convolutional ANNs in achieving impressive results with the game of Go.
Schmidhuber (2015) reviews applications of ANNs in reinforcement learning,
including applications of recurrent ANNs.
9.8

LSTD is due to Bradtke and Barto (see Bradtke, 1993, 1994; Bradtke and Barto,
1996; Bradtke, Ydstie, and Barto, 1994), and was further developed by Boyan(1999, 2002), Nedić and Bertsekas (2003), and Yu (2010). The incremental updateof the inverse matrix has been known at least since 1949 (Sherman and Morrison,1949). An extension of least-squares methods to control was introduced by
Lagoudakis and Parr (2003; Buşoniu, Lazaric, Ghavamzadeh, Munos, Babus̆ka,
and De Schutter, 2012).

9.9

Our discussion of memory-based function approximation is largely based on
the review of locally weighted learning by Atkeson, Moore, and Schaal (1997).
Atkeson (1992) discussed the use of locally weighted regression in memory-based
robot learning and supplied an extensive bibliography covering the history of
the idea. Stanfill and Waltz (1986) influentially argued for the importance ofmemory based methods in artificial intelligence, especially in light of parallelarchitectures then becoming available, such as the Connection Machine. Bairdand Klopf (1993) introduced a novel memory-based approach and used it as thefunction approximation method for Q-learning applied to the pole-balancing task.
Schaal and Atkeson (1994) applied locally weighted regression to a robot juggling
control problem, where it was used to learn a system model. Peng (1995) used
the pole-balancing task to experiment with several nearest-neighbor methods
for approximating value functions, policies, and environment models. Tadepalli
and Ok (1996) obtained promising results with locally-weighted linear regression
to learn a value function for a simulated automatic guided vehicle task. Bottou
and Vapnik (1992) demonstrated surprising efficiency of several local learningalgorithms compared to non-local algorithms in some pattern recognition tasks,
discussing the impact of local learning on generalization.
Bentley (1975) introduced k-d trees and reported observing average running
time of O(log n) for nearest neighbor search over n records. Friedman, Bentley,
and Finkel (1977) clarified the algorithm for nearest neighbor search with k-dtrees. Omohundro (1987) discussed efficiency gains possible with hierarchicaldata structures such as k-d-trees. Moore, Schneider, and Deng (1997) introduced
the use of k-d trees for efficient locally weighted regression.

9.10

The origin of kernel regression is the method of potential functions of Aizerman,
Braverman, and Rozonoer (1964). They likened the data to point electric chargesof various signs and magnitudes distributed over space. The resulting electric
potential over space produced by summing the potentials of the point charges
corresponded to the interpolated surface. In this analogy, the kernel function is
the potential of a point charge, which falls o↵ as the reciprocal of the distancefrom the charge. Connell and Utgo↵ (1987) applied an actor–critic method
to the pole-balancing task in which the critic approximated the value function

9.12. Summary

241

using kernel regression with an inverse-distance weighting. Predating widespreadinterest in kernel regression in machine learning, these authors did not use theterm kernel, but referred to “Shepard’s method” (Shepard, 1968). Other kernelbased approaches to reinforcement learning include those of Ormoneit and Sen(2002), Dietterich and Wang (2002), Xu, Xie, Hu, and Lu (2005), Taylor and Parr
(2009), Barreto, Precup, and Pineau (2011), and Bhat, Farias, and Moallemi(2012).
9.11

For Emphatic-TD methods, see the bibliographical notes to Section 11.8.

The earliest example we know of in which function approximation methods wereused for learning value functions was Samuel’s checkers player (1959, 1967). Samuelfollowed Shannon’s (1950) suggestion that a value function did not have to be exact tobe a useful guide to selecting moves in a game and that it might be approximated bya linear combination of features. In addition to linear function approximation, Samuelexperimented with lookup tables and hierarchical lookup tables called signature tables(Griffith, 1966, 1974; Page, 1977; Biermann, Fairfield, and Beres, 1982).
At about the same time as Samuel’s work, Bellman and Dreyfus (1959) proposed using
function approximation methods with DP. (It is tempting to think that Bellman andSamuel had some influence on one another, but we know of no reference to the other inthe work of either.) There is now a fairly extensive literature on function approximation
methods and DP, such as multigrid methods and methods using splines and orthogonal
polynomials (e.g., Bellman and Dreyfus, 1959; Bellman, Kalaba, and Kotkin, 1963; Daniel,
1976; Whitt, 1978; Reetz, 1977; Schweitzer and Seidmann, 1985; Chow and Tsitsiklis,
1991; Kushner and Dupuis, 1992; Rust, 1996).
Holland’s (1986) classifier system used a selective feature-match technique to generalize
evaluation information across state–action pairs. Each classifier matched a subset of states
having specified values for a subset of features, with the remaining features having arbitraryvalues (“wild cards”). These subsets were then used in a conventional state-aggregation
approach to function approximation. Holland’s idea was to use a genetic algorithm
to evolve a set of classifiers that collectively would implement a useful action-valuefunction. Holland’s ideas influenced the early research of the authors on reinforcement
learning, but we focused on di↵erent approaches to function approximation. As function
approximators, classifiers are limited in several ways. First, they are state-aggregation
methods, with concomitant limitations in scaling and in representing smooth functions
efficiently. In addition, the matching rules of classifiers can implement only aggregation
boundaries that are parallel to the feature axes. Perhaps the most important limitation of
conventional classifier systems is that the classifiers are learned via the genetic algorithm,
an evolutionary method. As we discussed in Chapter 1, there is available during learning
much more detailed information about how to learn than can be used by evolutionary
methods. This perspective led us to instead adapt supervised learning methods foruse in reinforcement learning, specifically gradient-descent and ANN methods. Thesedi↵erences between Holland’s approach and ours are not surprising because Holland’sideas were developed during a period when ANNs were generally regarded as being too
weak in computational power to be useful, whereas our work was at the beginning of242

Chapter 9: On-policy Prediction with Approximation

the period that saw widespread questioning of that conventional wisdom. There remain
many opportunities for combining aspects of these di↵erent approaches.
Christensen and Korf (1986) experimented with regression methods for modifying
coefficients of linear value function approximations in the game of chess. Chapman
and Kaelbling (1991) and Tan (1991) adapted decision-tree methods for learning valuefunctions. Explanation-based learning methods have also been adapted for learning
value functions, yielding compact representations (Yee, Saxena, Utgo↵, and Barto, 1990;
Dietterich and Flann, 1995).

Chapter 10

On-policy Control with
Approximation
In this chapter we return to the control problem, now with parametric approximation of
the action-value function q̂(s, a, w) ⇡ q⇤ (s, a), where w 2 Rd is a finite-dimensional weightvector. We continue to restrict attention to the on-policy case, leaving o↵-policy methods
to Chapter 11. The present chapter features the semi-gradient Sarsa algorithm, thenatural extension of semi-gradient TD(0) (last chapter) to action values and to on-policy
control. In the episodic case, the extension is straightforward, but in the continuing case
we have to take a few steps backward and re-examine how we have used discounting todefine an optimal policy. Surprisingly, once we have genuine function approximation we
have to give up discounting and switch to a new “average-reward” formulation of thecontrol problem, with new “di↵erential” value functions.
Starting first in the episodic case, we extend the function approximation ideas presented
in the last chapter from state values to action values. Then we extend them to control
following the general pattern of on-policy GPI, using "-greedy for action selection. We
show results for n-step linear Sarsa on the Mountain Car problem. Then we turn to thecontinuing case and repeat the development of these ideas for the average-reward casewith di↵erential values.

10.1

Episodic Semi-gradient Control

The extension of the semi-gradient prediction methods of Chapter 9 to action values isstraightforward. In this case it is the approximate action-value function, q̂ ⇡ q⇡ , that isrepresented as a parameterized functional form with weight vector w. Whereas before we
considered random training examples of the form St 7! Ut , now we consider examplesof the form St , At 7! Ut . The update target Ut can be any approximation of q⇡ (St , At ),including the usual backed-up values such as the full Monte Carlo return (Gt ) or anyof the n-step Sarsa returns (7.4). The general gradient-descent update for action-value

244

Chapter 10: On-policy Control with Approximation

prediction ish.
wt+1 = wt + ↵ Utiq̂(St , At , wt ) rq̂(St , At , wt ).

For example, the update for the one-step Sarsa method ishi.
wt+1 = wt + ↵ Rt+1 + q̂(St+1 , At+1 , wt ) q̂(St , At , wt ) rq̂(St , At , wt ).

(10.1)(10.2)

We call this method episodic semi-gradient one-step Sarsa. For a constant policy, thismethod converges in the same way that TD(0) does, with the same kind of error bound(9.14).
To form control methods, we need to couple such action-value prediction methods withtechniques for policy improvement and action selection. Suitable techniques applicable tocontinuous actions, or to actions from large discrete sets, are a topic of ongoing researchwith as yet no clear resolution. On the other hand, if the action set is discrete and not toolarge, then we can use the techniques already developed in previous chapters. That is, foreach possible action a available in the next state St+1 , we can compute q̂(St+1 , a, wt ) andthen find the greedy action A⇤t+1 = argmaxa q̂(St+1 , a, wt ). Policy improvement is thendone (in the on-policy case treated in this chapter) by changing the estimation policy to asoft approximation of the greedy policy such as the "-greedy policy. Actions are selectedaccording to this same policy. Pseudocode for the complete algorithm is given in the box.
Episodic Semi-gradient Sarsa for Estimating q̂ ⇡ q⇤
Input: a di↵erentiable action-value function parameterization q̂ : S ⇥ A ⇥ Rd ! R
Algorithm parameters: step size ↵ > 0, small " > 0
Initialize value-function weights w 2 Rd arbitrarily (e.g., w = 0)
Loop for each episode:
S, A
initial state and action of episode (e.g., "-greedy)
Loop for each step of episode:Take action A, observe R, S 0
If S 0 is terminal:⇥⇤w
w + ↵ R q̂(S, A, w) rq̂(S, A, w)
Go to next episode
Choose A0 as
of q̂(S 0 , ·, w) (e.g.,⇤"-greedy)⇥ a function
0w
w + ↵ R + q̂(S , A0 , w) q̂(S, A, w) rq̂(S, A, w)
S
S0
A
A0
Example 10.1: Mountain Car Task Consider the task of driving an underpowered
car up a steep mountain road, as suggested by the diagram in the upper left of Figure 10.1.
The difficulty is that gravity is stronger than the car’s engine, and even at full throttle
the car cannot accelerate up the steep slope. The only solution is to first move away fromthe goal and up the opposite slope on the left. Then, by applying full throttle the car10.1. Episodic Semi-gradient Control

Goal
Goal
Step 428

Episode 12
27

Ve
n

P os

7

i ti o

!. 0

P os

l oc

0

l oc

0
!1 .
2

i ty

i ty

.0 7

4

i ti o nVe

MOUNTAIN C AR

245

0 .6

Episode 104

Episode 1000

i ti o n

P osi ti o n

i ty

0

l oc

i ty

0

l oc

P os

Ve

l oc

0

120

Ve

i ty

104

P os

i ti o n

Ve

46

Episode 9000

Figure 10.1: The Mountain Car task (upper left panel) and the cost-to-go function( maxa q̂(s, a, w)) learned during one run.

can build up enough inertia to carry it up the steep slope even though it is slowing down
the whole way. This is a simple example of a continuous control task where things haveto get worse in a sense (farther from the goal) before they can get better. Many control
methodologies have great difficulties with tasks of this kind unless explicitly aided by ahuman designer.
The reward in this problem is 1 on all time steps until the car moves past its goal
position at the top of the mountain, which ends the episode. There are three possible
actions: full throttle forward (+1), full throttle reverse ( 1), and zero throttle (0). The
car moves according to a simplified physics. Its position, xt , and velocity, ẋt , are updatedby⇥⇤.
xt+1 = bound xt + ẋt+1⇥.
ẋt+1 = bound ẋt + 0.001At⇤
0.0025 cos(3xt ) ,where the bound operation enforces 1.2  xt+1  0.5 and 0.07  ẋt+1  0.07. In
addition, when xt+1 reached the left bound, ẋt+1 was reset to zero. When it reached
the right bound, the goal was reached and the episode was terminated. Each episodestarted from a random position xt 2 [ 0.6, 0.4) and zero velocity. To convert the twocontinuous state variables to binary features, we used grid-tilings as in Figure 9.9. We
used 8 tilings, with each tile covering 1/8th of the bounded distance in each dimension,

246

Chapter 10: On-policy Control with Approximation

and asymmetrical o↵sets as described in Section 9.5.4.1 The feature vectors x(s, a) createdby tile coding were then combined linearly with the parameter vector to approximate theaction-value function:dX.
q̂(s, a, w) = w> x(s, a) =wi · xi (s, a),(10.3)

i=1

for each pair of state, s, and action, a.
Figure 10.1 shows what typically happens while learning to solve this task with thisform of function approximation.2 Shown is the negative of the value function (the costto-go function) learned on a single run. The initial action values were all zero, which was
optimistic (all true values are negative in this task), causing extensive exploration to occur
even though the exploration parameter, ", was 0. This can be seen in the middle-top panelof the figure, labeled “Step 428”. At this time not even one episode had been completed,but the car has oscillated back and forth in the valley, following circular trajectories instate space. All the states visited frequently are valued worse than unexplored states,
because the actual rewards have been worse than what was (unrealistically) expected.
This continually drives the agent away from wherever it has been, to explore new states,
until a solution is found.
Figure 10.2 shows several learning curves for semi-gradient Sarsa on this problem, withvarious step sizes.
1000

Mountain Car

400↵ = 0.1/8↵ = 0.2/8

Steps per episodelog scaleaveraged over 100 runs

200

↵ = 0.5/8
1000Episode500

Figure 10.2: Mountain Car learning curves for the semi-gradient Sarsa method with tile-coding
function approximation and "-greedy action selection.

1 In particular, we used the tile-coding software, available at http://incompleteideas.net/tiles/
tiles3.html, with iht=IHT(4096) and tiles(iht,8,[8*x/(0.5+1.2),8*xdot/(0.07+0.07)],[A]) to get
the indices of the ones in the feature vector for state (x, xdot) and action A.
2 This data is actually from the “semi-gradient Sarsa( )” algorithm that we will not meet until
Chapter 12, but semi-gradient Sarsa would behave similarly.

10.2. Semi-gradient n-step Sarsa

10.2

247

Semi-gradient n-step Sarsa

We can obtain an n-step version of episodic semi-gradient Sarsa by using an n-step returnas the update target in the semi-gradient Sarsa update equation (10.1). The n-step return
immediately generalizes from its tabular form (7.4) to a function approximation form:.
Gt:t+n = Rt+1 + Rt+2 +· · ·+ n 1 Rt+n + n q̂(St+n , At+n , wt+n 1 ), t+n < T, (10.4).
with Gt:t+n = Gt if t + n T , as usual. The n-step update equation is.
wt+n = wt+n 1 + ↵ [Gt:t+nq̂(St , At , wt+n 1 )] rq̂(St , At , wt+n 1 ),0  t < T.
(10.5)

Complete pseudocode is given in the box below.
Episodic semi-gradient n-step Sarsa for estimating q̂ ⇡ q⇤ or q⇡
Input: a di↵erentiable action-value function parameterization q̂ : S ⇥ A ⇥ Rd ! R
Input: a policy ⇡ (if estimating q⇡ )Algorithm parameters: step size ↵ > 0, small " > 0, a positive integer n
Initialize value-function weights w 2 Rd arbitrarily (e.g., w = 0)
All store and access operations (St , At , and Rt ) can take their index mod n + 1
Loop for each episode:
Initialize and store S0 6= terminal
Select and store an action A0 ⇠ ⇡(·|S0 ) or "-greedy wrt q̂(S0 , ·, w)
T
1
Loop for t = 0, 1, 2, . . . :| If t < T , then:|Take action At|
Observe and store the next reward as Rt+1 and the next state as St+1|
If St+1 is terminal, then:
|
T
t+1
|
else:
|
Select and store At+1 ⇠ ⇡(·|St+1 ) or "-greedy wrt q̂(St+1 , ·, w)| ⌧t n + 1 (⌧ is the time whose estimate is being updated)| If ⌧ 0:
Pmin(⌧ +n,T ) i ⌧ 1|
GRi
i=⌧ +1|
If ⌧ + n < T , then G
G + n q̂(S⌧ +n , A⌧ +n , w)
(G⌧ :⌧ +n )|w
w + ↵ [G q̂(S⌧ , A⌧ , w)] rq̂(S⌧ , A⌧ , w)
Until ⌧ = T 1
As we have seen before, performance is best if an intermediate level of bootstrappingis used, corresponding to an n larger than 1. Figure 10.3 shows how this algorithm tends
to learn faster and obtain a better asymptotic performance at n = 8 than at n = 1 on the
Mountain Car task. Figure 10.4 shows the results of a more detailed study of the e↵ectof the parameters ↵ and n on the rate of learning on this task.

248

Chapter 10: On-policy Control with Approximation

1000

Mountain Car

400

Steps per episodelog scaleaveraged over 100 runs

200

n=1
n=8
100
0

500

EpisodeFigure 10.3: Performance of one-step vs 8-step semi-gradient Sarsa on the Mountain Car task.
Good step sizes were used: ↵ = 0.5/8 for n = 1 and ↵ = 0.3/8 for n = 8.

n=1

300

n=16
n=8
280

Mountain Car
Steps per episodeaveraged overfirst 50 episodes
and 100 runs

n=4
260

n=2
n=16

240

n=8
0

0.5

n=1

n=2

n=4

220

1

1.5

↵ × number of tilings (8)Figure 10.4: E↵ect of the ↵ and n on early performance of n-step semi-gradient Sarsa andtile-coding function approximation on the Mountain Car task. As usual, an intermediate level ofbootstrapping (n = 4) performed best. These results are for selected ↵ values, on a log scale,and then connected by straight lines. The standard errors ranged from 0.5 (less than the line
width) for n = 1 to about 4 for n = 16, so the main e↵ects are all statistically significant.

Exercise 10.1 We have not explicitly considered or given pseudocode for any Monte Carlo
methods in this chapter. What would they be like? Why is it reasonable not to give
pseudocode for them? How would they perform on the Mountain Car task?
⇤
Exercise 10.2 Give pseudocode for semi-gradient one-step Expected Sarsa for control. ⇤Exercise 10.3 Why do the results shown in Figure 10.4 have higher standard errors atlarge n than at small n?
⇤

10.3. Average Reward: A New Problem Setting for Continuing Tasks

10.3

249

Average Reward: A New Problem Setting forContinuing Tasks

We now introduce a third classical setting—alongside the episodic and discounted settings—
for formulating the goal in Markov decision problems (MDPs). Like the discounted
setting, the average reward setting applies to continuing problems, problems for which theinteraction between agent and environment goes on and on forever without termination
or start states. Unlike that setting, however, there is no discounting—the agent cares justas much about delayed rewards as it does about immediate reward. The average-reward
setting is one of the major settings commonly considered in the classical theory of dynamic
programming and less-commonly in reinforcement learning. As we discuss in the next
section, the discounted setting is problematic with function approximation, and thus theaverage-reward setting is needed to replace it.
In the average-reward setting, the quality of a policy ⇡ is defined as the average rate ofreward, or simply average reward, while following that policy, which we denote as r(⇡):h

1X.
r(⇡) = lim
E[Rt | S0 , A0:t 1 ⇠ ⇡]
h!1 h
t=1

= lim E[Rt | S0 , A0:t 1 ⇠ ⇡] ,
t!1
X
X
X
=
µ⇡ (s)⇡(a|s)p(s0 , r |s, a)r,s

a(10.6)(10.7)

s0,rwhere the expectations are conditioned on the initial state, S0 , and on the subsequentactions, A0 , A1 , . . . , At 1 , being taken according to ⇡. The second and third equations.
hold if the steady-state distribution, µ⇡ (s) = limt!1 Pr{St = s |A0:t 1 ⇠ ⇡}, exists andis independent of S0 , in other words, if the MDP is ergodic. In an ergodic MDP, thestarting state and any early decision made by the agent can have only a temporary e↵ect;in the long run the expectation of being in a state depends only on the policy and the
MDP transition probabilities. Ergodicity is sufficient but not necessary to guarantee theexistence of the limit in (10.6).
There are subtle distinctions that can be drawn between di↵erent kinds of optimality
in the undiscounted continuing case. Nevertheless, for most practical purposes it may
be adequate simply to order policies according to their average reward per time step,in other words, according to their r(⇡). This quantity is essentially the average reward
under ⇡, as suggested by (10.7), or the reward rate. In particular, we consider all policies
that attain the maximal value of r(⇡) to be optimal.
Note that the steady state distribution µ⇡ is the special distribution under which, if
you select actions according to ⇡, you remain in the same distribution. That is, for whichXXµ⇡ (s)⇡(a|s)p(s0 |s, a) = µ⇡ (s0 ).
(10.8)
sa

250

Chapter 10: On-policy Control with Approximation

In the average-reward setting, returns are defined in terms of di↵erences between
rewards and the average reward:.
Gt = Rt+1 r(⇡) + Rt+2 r(⇡) + Rt+3 r(⇡) + · · · .
(10.9)

This is known as the di↵erential return, and the corresponding value functions are
known as di↵erential value functions. Di↵erential value functions are defined in termsof the new return just as conventional value functions were defined in terms of the.
discounted return; thus we will use the same notation, v⇡ (s) = E⇡[Gt |St = s] and.
q⇡ (s, a) = E⇡[Gt |St = s, At = a] (similarly for v⇤ and q⇤ ), for di↵erential value functions.
Di↵erential value functions also have Bellman equations, just slightly di↵erent from those
we have seen earlier. We simply remove all s and replace all rewards by the di↵erence
between the reward and the true average reward:hiXXv⇡ (s) =⇡(a|s)p(s0 , r |s, a) r r(⇡) + v⇡ (s0 ) ,aq⇡ (s, a) =Xr,s0v⇤ (s) = maxaq⇤ (s, a) =r,s0hp(s0 , r |s, a) r

X
r,s0

X
r,s0

r(⇡) +
hp(s0 , r |s, a) rhp(s0 , r |s, a) rXa0i⇡(a0 |s0 )q⇡ (s0 , a0 ) ,imax r(⇡) + v⇤ (s0 ) , and⇡max r(⇡) + maxq⇤ (s0 , a0 )0⇡

a(cf. (3.14), Exercise 3.17, (3.19), and (3.20)).
There is also a di↵erential form of the two TD errors:.
v̂(St ,wt ),t = Rt+1 R̄t + v̂(St+1 ,wt )andt.
= Rt+1 R̄t + q̂(St+1 , At+1 , wt )q̂(St , At , wt ),i(10.10)(10.11)

where R̄t is an estimate at time t of the average reward r(⇡). With these alternate
definitions, most of our algorithms and many theoretical results carry through to theaverage-reward setting without change.
For example, an average reward version of semi-gradient Sarsa could be defined just asin (10.2) except with the di↵erential version of the TD error. That is, by.
wt+1 = wt + ↵ t rq̂(St , At , wt ),(10.12)
with t given by (10.11). Pseudocode for a complete algorithm is given in the box on thenext page. One limitation of this algorithm is that it does not converge to the di↵erential
values but to the di↵erential values plus an arbitrary o↵set. Notice that the Bellman
equations and TD errors given above are una↵ected if all the values are shifted by thesame amount. Thus, the o↵set may not matter in practice. How this algorithm could bechanged to eliminate the o↵set is an interesting question for future research.
Exercise 10.4 Give pseudocode for a di↵erential version of semi-gradient Q-learning. ⇤Exercise 10.5 What equations are needed (beyond 10.10) to specify the di↵erential
version of TD(0)?
⇤10.3. Average Reward: A New Problem Setting for Continuing Tasks

251

Di↵erential semi-gradient Sarsa for estimating q̂ ⇡ q⇤
Input: a di↵erentiable action-value function parameterization q̂ : S ⇥ A ⇥ Rd ! R
Algorithm parameters: step sizes ↵, > 0, small " > 0
Initialize value-function weights w 2 Rd arbitrarily (e.g., w = 0)
Initialize average reward estimate R̄ 2 R arbitrarily (e.g., R̄ = 0)
Initialize state S, and action A
Loop for each step:
Take action A, observe R, S 0
Choose A0 as a function of q̂(S 0 , ·, w) (e.g., "-greedy)
R R̄ + q̂(S 0 , A0 , w) q̂(S, A, w)
R̄
R̄ +
w
w + ↵ rq̂(S, A, w)
S
S0
A
A0
Exercise 10.6 Suppose there is an MDP that under any policy produces the deterministic
sequence of rewards +1, 0, +1, 0, +1, 0, . . . going on forever. Technically, this violates
ergodicity; there is no stationary limiting distribution µ⇡ and the limit (10.7) does notexist. Nevertheless, the average reward (10.6) is well defined. What is it? Now consider
two states in this MDP. From A, the reward sequence is exactly as described above,
starting with a +1, whereas, from B, the reward sequence starts with a 0 and thencontinues with +1, 0, +1, 0, . . .. We would like to compute the di↵erential values of A and
B. Unfortunately, the di↵erential return (10.9) is not well defined when starting fromthese states as the implicit limit does not exist. To repair this, one could alternatively
define the di↵erential value of a state ashX.
v⇡ (s) = lim lim!1 h!1t=0t⇣

E⇡[Rt+1 |S0 = s]⌘r(⇡) .

Under this definition, what are the di↵erential values of states A and B?

(10.13)⇤

Exercise 10.7 Consider a Markov reward process consisting of a ring of three states A, B,
and C, with state transitions going deterministically around the ring. A reward of +1 isreceived upon arrival in A and otherwise the reward is 0. What are the di↵erential valuesof the three states, using (10.13)?
⇤
Exercise 10.8 The pseudocode in the box on page 251 updates R̄t using t as an error
rather than simply Rt+1 R̄t . Both errors work, but using t is better. To see why,
consider the ring MRP of three states from Exercise 10.7. The estimate of the averagereward should tend towards its true value of 13 . Suppose it was already there and washeld stuck there. What would the sequence of Rt+1 R̄t errors be? What would thesequence of t errors be (using Equation 10.10)? Which error sequence would produce
a more stable estimate of the average reward if the estimate were allowed to change in
response to the errors? Why?
⇤

252

Chapter 10: On-policy Control with Approximation

Example 10.2: An Access-Control Queuing Task This is a decision task involving
access control to a set of 10 servers. Customers of four di↵erent priorities arrive at asingle queue. If given access to a server, the customers pay a reward of 1, 2, 4, or 8 tothe server, depending on their priority, with higher priority customers paying more. In
each time step, the customer at the head of the queue is either accepted (assigned to oneof the servers) or rejected (removed from the queue, with a reward of zero). In either
case, on the next time step the next customer in the queue is considered. The queue
never empties, and the priorities of the customers in the queue are uniformly randomly
distributed. Of course a customer cannot be served if there is no free server; the customer
is always rejected in this case. Each busy server becomes free with probability p = 0.06
on each time step. Although we have just described them for definiteness, let us assume
the statistics of arrivals and departures are unknown. The task is to decide on each stepwhether to accept or reject the next customer, on the basis of his priority and the numberof free servers, so as to maximize long-term reward without discounting.
In this example we consider a tabular solution to this problem. Although there is no
generalization between states, we can still consider it in the general function approximation
setting as this setting generalizes the tabular setting. Thus we have a di↵erential actionvalue estimate for each pair of state (number of free servers and priority of the customer
at the head of the queue) and action (accept or reject). Figure 10.5 shows the solution
found by di↵erential semi-gradient Sarsa with parameters ↵ = 0.01, = 0.01, and " = 0.1.
The initial action values and R̄ were zero.
1

Priority

POLICYREJECT

2

POLICYACCEPT

4
8
1

2

3

4

5

6

7

8

9

10

Number of free servers
priority 8

7 10
5V ALUEFUNCTION

4prioritypriority
8
priority 4

5

Diﬀerential 0
value
Value
of of 0
bestbest
actionactionpriority 2priority 2priority 1priority 1

VALUE
FUNCTION

!5 -5
-10

!10

0

1

2

3

4

5

6

7

8

9

10

Number of free servers!15
2 function
3
4 found5 by
6 di↵erential
7
8
9semi-gradient10
Figure 10.5: The policy 0and 1value
one-step Sarsa
on the access-control queuing task after
2 million
steps.
The drop on the right of the graph
Numberof freeservers
is probably due to insufficient data; many of these states were never experienced. The valuelearned for R̄ was about 2.31. (Note that priority 1 here is the lowest priority.)

10.4. Deprecating the Discounted Setting

10.4

253

Deprecating the Discounted Setting

The continuing, discounted problem formulation has been very useful in the tabular case,in which the returns from each state can be separately identified and averaged. But in theapproximate case it is questionable whether one should ever use this problem formulation.
To see why, consider an infinite sequence of returns with no beginning or end, and noclearly identified states. The states might be represented only by feature vectors, which
may do little to distinguish the states from each other. As a special case, all of the feature
vectors may be the same. Thus one really has only the reward sequence (and the actions),
and performance has to be assessed purely from these. How could it be done? One way
is by averaging the rewards over a long interval—this is the idea of the average-reward
setting. How could discounting be used? Well, for each time step we could measure
the discounted return. Some returns would be small and some big, so again we would
have to average them over a sufficiently large time interval. In the continuing setting
there are no starts and ends, and no special time steps, so there is nothing else that
could be done. However, if you do this, it turns out that the average of the discounted
returns is proportional to the average reward. In fact, for policy ⇡, the average of the
discounted returns is always r(⇡)/(1), that is, it is essentially the average reward,r(⇡). In particular, the ordering of all policies in the average discounted return setting
would be exactly the same as in the average-reward setting. The discount rate thus has
no e↵ect on the problem formulation. It could in fact be zero and the ranking would beunchanged.
This surprising fact is proven in the box on the next page, but the basic idea canbe seen via a symmetry argument. Each time step is exactly the same as every other.
With discounting, every reward will appear exactly once in each position in some return.
The tth reward will appear undiscounted in the t 1st return, discounted once in thet 2nd return, and discounted 999 times in the t 1000th return. The weight on thetth reward is thus 1 + + 2 + 3 + · · · = 1/(1). Because all states are the same,
they are all weighted by this, and thus the average of the returns will be this times theaverage reward, or r(⇡)/(1).
This example and the more general argument in the box show that if we optimized
discounted value over the on-policy distribution, then the e↵ect would be identical tooptimizing undiscounted average reward; the actual value of would have no e↵ect. Thisstrongly suggests that discounting has no role to play in the definition of the control
problem with function approximation. One can nevertheless go ahead and use discounting
in solution methods. The discounting parameter changes from a problem parameterto a solution method parameter! Unfortunately, discounting algorithms with function
approximation do not optimize discounted value over the on-policy distribution, and thus
are not guaranteed to optimize average reward.
The root cause of the difficulties with the discounted control setting is that withfunction approximation we have lost the policy improvement theorem (Section 4.2). It is
no longer true that if we change the policy to improve the discounted value of one statethen we are guaranteed to have improved the overall policy in any useful sense. That
guarantee was key to the theory of our reinforcement learning control methods. With

254

Chapter 10: On-policy Control with Approximation

The Futility of Discounting in Continuing Problems
Perhaps discounting can be saved by choosing an objective that sums discounted
values over the distribution with which states occur under the policy:
X
J(⇡) =µ⇡ (s)v⇡ (s)(where v⇡ is the discounted value function)s=Xµ⇡ (s)s= r(⇡) +X

X⇡(a|s)aµ⇡ (s)s= r(⇡) +

X

X

X

s0

r

⇡(a|s)v⇡ (s0 )X

p(s0 , r |s, a) [r + v⇡ (s0 )](Bellman Eq.)

XX

(from (10.7))

s0

a

s0

= r(⇡) +XXµ⇡ (s)sXav⇡ (s0 )µ⇡ (s0 )rp(s0 , r |s, a) v⇡ (s0 )⇡(a|s)p(s0 |s, a)(from (3.4))(from (10.8))

s0= r(⇡) + J(⇡)
= r(⇡) + r(⇡) +

2

J(⇡)

= r(⇡) + r(⇡) +
1
=
r(⇡).
1

2

r(⇡) +

3r(⇡) + · · ·The proposed discounted objective orders policies identically to the undiscounted(average reward) objective. The discount rate does not influence the ordering!

function approximation we have lost it!
In fact, the lack of a policy improvement theorem is also a theoretical lacuna for the
total-episodic and average-reward settings. Once we introduce function approximation
we can no longer guarantee improvement for any setting. In Chapter 13 we introduce analternative class of reinforcement learning algorithms based on parameterized policies,
and there we have a theoretical guarantee called the “policy-gradient theorem” whichplays a similar role as the policy improvement theorem. But for methods that learn
action values we seem to be currently without a local improvement guarantee (possibly
the approach taken by Perkins and Precup (2003) may provide a part of the answer). We
do know that "-greedification may sometimes result in an inferior policy, as policies may
chatter among good policies rather than converge (Gordon, 1996a). This is an area withmultiple open theoretical questions.

10.5. Di↵erential Semi-gradient n-step Sarsa

255

Di↵erential Semi-gradient n-step Sarsa

10.5

In order to generalize to n-step bootstrapping, we need an n-step version of the TD error.
We begin by generalizing the n-step return (7.4) to its di↵erential form, with function
approximation:.
Gt:t+n = Rt+1 R̄t+n 1 + · · · + Rt+n R̄t+n 1 + q̂(St+n , At+n , wt+n 1 ),where R̄ is an estimate of r(⇡), n
1, and t + n < T . If t + n.
Gt:t+n = Gt as usual. The n-step TD error is then.

t = Gt:t+n(10.14)T , then we defineq̂(St , At , w),(10.15)after which we can apply our usual semi-gradient Sarsa update (10.12). Pseudocode forthe complete algorithm is given in the box.
Di↵erential semi-gradient n-step Sarsa for estimating q̂ ⇡ q⇡ or q⇤
Input: a di↵erentiable function q̂ : S ⇥ A ⇥ Rd ! R, a policy ⇡Initialize value-function weights w 2 Rd arbitrarily (e.g., w = 0)
Initialize average-reward estimate R̄ 2 R arbitrarily (e.g., R̄ = 0)
Algorithm parameters: step sizes ↵, > 0, small " > 0, a positive integer n
All store and access operations (St , At , and Rt ) can take their index mod n + 1
Initialize and store S0 and A0
Loop for each step, t = 0, 1, 2, . . . :Take action At
Observe and store the next reward as Rt+1 and the next state as St+1
Select and store an action At+1 ⇠ ⇡(·|St+1 ), or "-greedy wrt q̂(St+1 , ·, w)⌧
t n + 1 (⌧ is the time whose estimate is being updated)
If ⌧ 0:P⌧ +nR̄) + q̂(S⌧ +n , A⌧ +n , w) q̂(S⌧ , A⌧ , w)i=⌧ +1 (Ri
R̄
R̄ +
w
w + ↵ rq̂(S⌧ , A⌧ , w)
Exercise 10.9 In the di↵erential semi-gradient n-step Sarsa algorithm, the step-size
parameter on the average reward, , needs to be quite small so that R̄ becomes a goodlong-term estimate of the average reward. Unfortunately, R̄ will then be biased by its
initial value for many steps, which may make learning inefficient. Alternatively, one could
use a sample average of the observed rewards for R̄. That would initially adapt rapidly
but in the long run would also adapt slowly. As the policy slowly changed, R̄ would alsochange; the potential for such long-term nonstationarity makes sample-average methods
ill-suited. In fact, the step-size parameter on the average reward is a perfect place to use
the unbiased constant-step-size trick from Exercise 2.7. Describe the specific changes
needed to the boxed algorithm for di↵erential semi-gradient n-step Sarsa to use thistrick.
⇤

256

10.6

Chapter 10: On-policy Control with Approximation

Summary

In this chapter we have extended the ideas of parameterized function approximation andsemi-gradient descent, introduced in the previous chapter, to control. The extension isimmediate for the episodic case, but for the continuing case we have to introduce a whole
new problem formulation based on maximizing the average reward setting per time step.
Surprisingly, the discounted formulation cannot be carried over to control in the presenceof approximations. In the approximate case most policies cannot be represented by avalue function. The arbitrary policies that remain need to be ranked, and the scalar
average reward r(⇡) provides an e↵ective way to do this.
The average reward formulation involves new di↵erential versions of value functions,
Bellman equations, and TD errors, but all of these parallel the old ones, and theconceptual changes are small. There is also a new parallel set of di↵erential algorithms
for the average-reward case.

Bibliographical and Historical Remarks
10.1

Semi-gradient Sarsa with function approximation was first explored by Rummeryand Niranjan (1994). Linear semi-gradient Sarsa with "-greedy action selection
does not converge in the usual sense, but does enter a bounded region near
the best solution (Gordon, 1996a, 2001). Precup and Perkins (2003) showed
convergence in a di↵erentiable action selection setting. See also Perkins andPendrith (2002) and Melo, Meyn, and Ribeiro (2008). The mountain–car example
is based on a similar task studied by Moore (1990), but the exact form used hereis from Sutton (1996).

10.2

Episodic n-step semi-gradient Sarsa is based on the forward Sarsa( ) algorithmof van Seijen (2016). The empirical results shown here are new to the second
edition of this text.

10.3

The average-reward formulation has been described for dynamic programming(e.g., Puterman, 1994) and from the point of view of reinforcement learning(Mahadevan, 1996; Tadepalli and Ok, 1994; Bertsekas and Tsitsiklis, 1996;
Tsitsiklis and Van Roy, 1999). The algorithm described here is the on-policy
analog of the “R-learning” algorithm introduced by Schwartz (1993). The nameR-learning was probably meant to be the alphabetic successor to Q-learning,
but we prefer to think of it as a reference to the learning of di↵erential or
relative values. The access-control queuing example was suggested by the workof Carlström and Nordström (1997).

10.4

The recognition of the limitations of discounting as a formulation of the reinforcement learning problem with function approximation became apparent tothe authors shortly after the publication of the first edition of this text. Singh,
Jaakkola, and Jordan (1994) may have been the first to observe it in print.

Chapter 11*O↵-policy Methods with
Approximation
This book has treated on-policy and o↵-policy learning methods since Chapter 5 primarily
as two alternative ways of handling the conflict between exploitation and exploration
inherent in learning forms of generalized policy iteration. The two chapters preceding thishave treated the on-policy case with function approximation, and in this chapter we treat
the o↵ -policy case with function approximation. The extension to function approximation
turns out to be significantly di↵erent and harder for o↵-policy learning than it is foron-policy learning. The tabular o↵-policy methods developed in Chapters 6 and 7 readilyextend to semi-gradient algorithms, but these algorithms do not converge as robustly as
they do under on-policy training. In this chapter we explore the convergence problems,
take a closer look at the theory of linear function approximation, introduce a notion oflearnability, and then discuss new algorithms with stronger convergence guarantees for theo↵-policy case. In the end we will have improved methods, but the theoretical results will
not be as strong, nor the empirical results as satisfying, as they are for on-policy learning.
Along the way, we will gain a deeper understanding of approximation in reinforcement
learning for on-policy learning as well as o↵-policy learning.
Recall that in o↵-policy learning we seek to learn a value function for a target policy⇡, given data due to a di↵erent behavior policy b. In the prediction case, both policies
are static and given, and we seek to learn either state values v̂ ⇡ v⇡ or action valuesq̂ ⇡ q⇡ . In the control case, action values are learned, and both policies typically change
during learning—⇡ being the greedy policy with respect to q̂, and b being something
more exploratory such as the "-greedy policy with respect to q̂.
The challenge of o↵-policy learning can be divided into two parts, one that arises inthe tabular case and one that arises only with function approximation. The first partof the challenge has to do with the target of the update (not to be confused with thetarget policy), and the second part has to do with the distribution of the updates. The
techniques related to importance sampling developed in Chapters 5 and 7 deal withthe first part; these may increase variance but are needed in all successful algorithms,

258

Chapter 11: O↵-policy Methods with Approximation

tabular and approximate. The extension of these techniques to function approximation
are quickly dealt with in the first section of this chapter.
Something more is needed for the second part of the challenge of o↵-policy learningwith function approximation because the distribution of updates in the o↵-policy case is
not according to the on-policy distribution. The on-policy distribution is important tothe stability of semi-gradient methods. Two general approaches have been explored todeal with this. One is to use importance sampling methods again, this time to warp the
update distribution back to the on-policy distribution, so that semi-gradient methods
are guaranteed to converge (in the linear case). The other is to develop true gradientmethods that do not rely on any special distribution for stability. We present methods
based on both approaches. This is a cutting-edge research area, and it is not clear whichof these approaches is most e↵ective in practice.

11.1

Semi-gradient Methods

We begin by describing how the methods developed in earlier chapters for the o↵policy case extend readily to function approximation as semi-gradient methods. Thesemethods address the first part of the challenge of o↵-policy learning (changing the updatetargets) but not the second part (changing the update distribution). Accordingly, thesemethods may diverge in some cases, and in that sense are not sound, but still they
are often successfully used. Remember that these methods are guaranteed stable andasymptotically unbiased for the tabular case, which corresponds to a special case offunction approximation. So it may still be possible to combine them with feature selection
methods in such a way that the combined system could be assured stable. In any event,these methods are simple and thus a good place to start.
In Chapter 7 we described a variety of tabular o↵-policy algorithms. To convert themto semi-gradient form, we simply replace the update to an array (V or Q) to an update
to a weight vector (w), using the approximate value function (v̂ or q̂) and its gradient.
Many of these algorithms use the per-step importance sampling ratio:⇡(At |St ).
⇢t = ⇢t:t =.
b(At |St )(11.1)For example, the one-step, state-value algorithm is semi-gradient o↵-policy TD(0), which
is just like the corresponding on-policy algorithm (page 203) except for the addition of⇢t :.
wt+1 = wt + ↵⇢t t rv̂(St ,wt ),(11.2)
where t is defined appropriately depending on whether the problem is episodic anddiscounted, or continuing and undiscounted using average reward:.
v̂(St ,wt ), or(11.3)
t = Rt+1 + v̂(St+1 ,wt ).

t = Rt+1

R̄t + v̂(St+1 ,wt )v̂(St ,wt ).

(11.4)

11.1. Semi-gradient Methods259

For action values, the one-step algorithm is semi-gradient Expected Sarsa:.
wt+1 = wt + ↵ t rq̂(St , At , wt ), withX.
⇡(a|St+1 )q̂(St+1 , a, wt ) q̂(St , At , wt ), ort = Rt+1 +a.

t = Rt+1

R̄t +X⇡(a|St+1 )q̂(St+1 , a, wt )q̂(St , At , wt ).

(11.5)(episodic)(continuing)aNote that this algorithm does not use importance sampling. In the tabular case it is clearthat this is appropriate because the only sample action is At , and in learning its value we
do not have to consider any other actions. With function approximation it is less clear
because we might want to weight di↵erent state–action pairs di↵erently once they allcontribute to the same overall approximation. Proper resolution of this issue awaits amore thorough understanding of the theory of function approximation in reinforcement
learning.
In the multi-step generalizations of these algorithms, both the state-value and actionvalue algorithms involve importance sampling. The n-step version of semi-gradient Sarsa
is.
wt+n = wt+n 1 +↵⇢t+1 · · · ⇢t+n [Gt:t+n q̂(St , At , wt+n 1 )] rq̂(St , At , wt+n 1 ) (11.6)
with.
Gt:t+n = Rt+1 + · · · + n 1 Rt+n + n q̂(St+n , At+n , wt+n 1 ), or(episodic).
Gt:t+n = Rt+1 R̄t + · · · + Rt+n R̄t+n 1 + q̂(St+n , At+n , wt+n 1 ), (continuing)

where here we are being slightly informal in our treatment of the ends of episodes. In thefirst equation, the ⇢k s for k T (where T is the last time step of the episode) should betaken to be 1, and Gt:t+n should be taken to be Gt if t + n T .
Recall that we also presented in Chapter 7 an o↵-policy algorithm that does not involve
importance sampling at all: the n-step tree-backup algorithm. Here is its semi-gradientversion:.
wt+n = wt+n 1 + ↵ [Gt:t+n q̂(St , At , wt+n 1 )] rq̂(St , At , wt+n 1 ),(11.7)t+nk
X1
Y.
Gt:t+n = q̂(St , At , wt+n 1 ) +
k
k=t

i=t+1

⇡(Ai |Si ),(11.8)

with t as defined at the top of this page for Expected Sarsa. We also defined in Chapter 7
an algorithm that unifies all action-value algorithms: n-step Q( ). We leave the semigradient form of that algorithm, and also of the n-step state-value algorithm, as exercisesfor the reader.
Exercise 11.1 Convert the equation of n-step o↵-policy TD (7.9) to semi-gradient form.
Give accompanying definitions of the return for both the episodic and continuing cases. ⇤⇤Exercise 11.2 Convert the equations of n-step Q( ) (7.11 and 7.17) to semi-gradientform. Give definitions that cover both the episodic and continuing cases.
⇤

260

Chapter 11: O↵-policy Methods with Approximation

11.2

Examples of O↵-policy Divergence

In this section we begin to discuss the second part of the challenge of o↵-policy learningwith function approximation—that the distribution of updates does not match the onpolicy distribution. We describe some instructive counterexamples to o↵-policy learning—cases where semi-gradient and other simple algorithms are unstable and diverge.
To establish intuitions, it is best to consider first a very simple example. Suppose,
perhaps as part of a larger MDP, there are two states whose estimated values are of
the functional form w and 2w, where the parameter vector w consists of only a single
component w. This occurs under linear function approximation if the feature vectors
for the two states are each simple numbers (single-component vectors), in this case 1
and 2. In the first state, only one action is available, and it results deterministically in atransition to the second state with a reward of 0:

2w

0

2w

where the expressions inside the two circles indicate the two state’s values.
Suppose initially w = 10. The transition will then be from a state of estimated value
10 to a state of estimated value 20. It will look like a good transition, and w will be
increased to raise the first state’s estimated value. If is nearly 1, then the TD error will
be nearly 10, and, if ↵ = 0.1, then w will be increased to nearly 11 in trying to reduce theTD error. However, the second state’s estimated value will also be increased, to nearly
22. If the transition occurs again, then it will be from a state of estimated value ⇡11 toa state of estimated value ⇡22, for a TD error of ⇡11—larger, not smaller, than before.
It will look even more like the first state is undervalued, and its value will be increased
again, this time to ⇡12.1. This looks bad, and in fact with further updates w will diverge
to infinity.
To see this definitively we have to look more carefully at the sequence of updates. The
TD error on a transition between the two states is
t = Rt+1 +v̂(St+1 ,wt )v̂(St ,wt ) = 0 + 2wt

wt = (21)wt ,and the o↵-policy semi-gradient TD(0) update (from (11.2)) is
wt+1 = wt + ↵⇢t t rv̂(St ,wt ) = wt + ↵ · 1 · (21)wt · 1 = 1 + ↵(21) wt .

Note that the importance sampling ratio, ⇢t , is 1 on this transition because there is
only one action available from the first state, so its probabilities of being taken under
the target and behavior policies must both be 1. In the final update above, the newparameter is the old parameter times a scalar constant, 1 + ↵(21). If this constant isgreater than 1, then the system is unstable and w will go to positive or negative infinity
depending on its initial value. Here this constant is greater than 1 whenever > 0.5.
Note that stability does not depend on the specific step size, as long as ↵ > 0. Smaller orlarger step sizes would a↵ect the rate at which w goes to infinity, but not whether it goes
there or not.
Key to this example is that the one transition occurs repeatedly without w being
updated on other transitions. This is possible under o↵-policy training because the11.2. Examples of O↵-policy Divergence261

behavior policy might select actions on those other transitions which the target policy
never would. For these transitions, ⇢t would be zero and no update would be made.
Under on-policy training, however, ⇢t is always one. Each time there is a transition fromthe w state to the 2w state, increasing w, there would also have to be a transition outof the 2w state. That transition would reduce w, unless it were to a state whose value
was higher (because < 1) than 2w, and then that state would have to be followed by astate of even higher value, or else again w would be reduced. Each state can support theone before only by creating a higher expectation. Eventually the piper must be paid. In
the on-policy case the promise of future reward must be kept and the system is kept incheck. But in the o↵-policy case, a promise can be made and then, after taking an actionthat the target policy never would, forgotten and forgiven.
This simple example communicates much of the reason why o↵-policy training can leadto divergence, but it is not completely convincing because it is not complete—it is just afragment of a complete MDP. Can there really be a complete system with instability? A
simple complete example of divergence is Baird’s counterexample. Consider the episodic
seven-state, two-action MDP shown in Figure 11.1. The dashed action takes the system
to one of the six upper states with equal probability, whereas the solid action takes thesystem to the seventh state. The behavior policy b selects the dashed and solid actions
with probabilities 67 and 17 , so that the next-state distribution under it is uniform (thesame for all nonterminal states), which is also the starting distribution for each episode.
The target policy ⇡ always takes the solid action, and so the on-policy distribution (for ⇡)is concentrated in the seventh state. The reward is zero on all transitions. The discountrate is = 0.99.
Consider estimating the state-value under the linear parameterization indicated by
the expression shown in each state circle. For example, the estimated value of theleftmost state is 2w1 + w8 , where the subscript corresponds to the component of the

2w1 +w8

2w2 +w8

2w3 +w8

2w4 +w8

2w5 +w8

2w6 +w8⇡(solid|·) = 1b(dashed|·) = 6/7

w7 +2w8b(solid|·) = 1/7= 0.99Figure 11.1: Baird’s counterexample. The approximate state-value function for this Markov
process is of the form shown by the linear expressions inside each state. The solid action usually
results in the seventh state, and the dashed action usually results in one of the other six states,
each with equal probability. The reward is always zero.

262

Chapter 11: O↵-policy Methods with Approximation

overall weight vector w 2 R8 ; this corresponds to a feature vector for the first state
being x(1) = (2, 0, 0, 0, 0, 0, 0, 1)> . The reward is zero on all transitions, so the true valuefunction is v⇡ (s) = 0, for all s, which can be exactly approximated if w = 0. In fact,
there are many solutions, as there are more components to the weight vector (8) thanthere are nonterminal states (7). Moreover, the set of feature vectors, {x(s) : s 2 S}, isa linearly independent set. In all these ways this task seems a favorable case for linear
function approximation.
If we apply semi-gradient TD(0) to this problem (11.2), then the weights diverge
to infinity, as shown in Figure 11.2 (left). The instability occurs for any positive stepsize, no matter how small. In fact, it even occurs if an expected update is done as in
dynamic programming (DP), as shown in Figure 11.2 (right). That is, if the weightvector, wk , is updated for all states at the same time in a semi-gradient way, using theDP (expectation-based) target:⌘↵ X⇣.
wk+1 = wk +E⇡[Rt+1 + v̂(St+1 ,wk ) | St = s] v̂(s,wk ) rv̂(s,wk ). (11.9)|S| s
In this case, there is no randomness and no asynchrony, just as in a classical DP update.
The method is conventional except in its use of semi-gradient function approximation.
Yet still the system is unstable.
If we alter just the distribution of DP updates in Baird’s counterexample, from theuniform distribution to the on-policy distribution (which generally requires asynchronous
updating), then convergence is guaranteed to a solution with error bounded by (9.14).
This example is striking because the TD and DP methods used are arguably the simplest

Semi-gradient Off-policy TD

Semi-gradient DP

300

w8

w8

200

w 1 – w6

w 1 – w6

w7

w7

100

10
1

0

Steps

1000

0

Sweeps1000

Figure 11.2: Demonstration of instability on Baird’s counterexample. Shown are the evolutionof the components of the parameter vector w of the two semi-gradient algorithms. The step size
was ↵ = 0.01, and the initial weights were w = (1, 1, 1, 1, 1, 1, 10, 1)> .

11.2. Examples of O↵-policy Divergence263

and best-understood bootstrapping methods, and the linear, semi-descent method used isarguably the simplest and best-understood kind of function approximation. The example
shows that even the simplest combination of bootstrapping and function approximation
can be unstable if the updates are not done according to the on-policy distribution.
There are also counterexamples similar to Baird’s showing divergence for Q-learning.
This is cause for concern because otherwise Q-learning has the best convergence guarantees
of all control methods. Considerable e↵ort has gone into trying to find a remedy tothis problem or to obtain some weaker, but still workable, guarantee. For example, it
may be possible to guarantee convergence of Q-learning as long as the behavior policy is
sufficiently close to the target policy, for example, when it is the "-greedy policy. To thebest of our knowledge, Q-learning has never been found to diverge in this case, but there
has been no theoretical analysis. In the rest of this section we present several other ideasthat have been explored.
Suppose that instead of taking just a step toward the expected one-step return on each
iteration, as in Baird’s counterexample, we actually change the value function all the way
to the best, least-squares approximation. Would this solve the instability problem? Of
course it would if the feature vectors, {x(s) : s 2 S}, formed a linearly independent set,as they do in Baird’s counterexample, because then exact approximation is possible oneach iteration and the method reduces to standard tabular DP. But of course the point
here is to consider the case when an exact solution is not possible. In this case stability
is not guaranteed even when forming the best approximation at each iteration, as shownin the example.
Example 11.1: Tsitsiklis and Van Roy’s Counterexample This example showsthat linear function approximation would not work with DP even if the least-squares
solution was found at each step. The counterexample is formedby extending the w-to-2w example (from earlier in this section)
1 "with a terminal state, as shown to the right. As before, theestimated value of the first state is w, and the estimated valueof the second state is 2w. The reward is zero on all transitions,w
2w
so the true values are zero at both states, which is exactly
representable with w = 0. If we set wk+1 at each step so"
as to minimize the VE between the estimated value and the
expected one-step return, then we have
X⇣
⇥
⇤ ⌘2
wk+1 = argminv̂(s,w) E⇡ Rt+1 + v̂(St+1 ,wk ) St = s
w2R

=

s2S

argmin w
w2R

=

6

4"
5

2wk

2

+ 2w

(1

") 2wk2

wk .

The sequence {wk } diverges when

(11.10)> 6 54" and w0 6= 0.

264

Chapter 11: O↵-policy Methods with Approximation

Another way to try to prevent instability is to use special methods for function
approximation. In particular, stability is guaranteed for function approximation methods
that do not extrapolate from the observed targets. These methods, called averagers,
include nearest neighbor methods and locally weighted regression, but not popularmethods such as tile coding and artificial neural networks (ANNs).
Exercise 11.3 (programming) Apply one-step semi-gradient Q-learning to Baird’s counterexample and show empirically that its weights diverge.
⇤

11.3

The Deadly Triad

Our discussion so far can be summarized by saying that the danger of instability anddivergence arises whenever we combine all of the following three elements, making up
what we call the deadly triad :Function approximation A powerful, scalable way of generalizing from a state spacemuch larger than the memory and computational resources (e.g., linear function
approximation or ANNs).
Bootstrapping Update targets that include existing estimates (as in dynamic programming or TD methods) rather than relying exclusively on actual rewards andcomplete returns (as in MC methods).
O↵-policy training Training on a distribution of transitions other than that producedby the target policy. Sweeping through the state space and updating all states
uniformly, as in dynamic programming, does not respect the target policy and is
an example of o↵-policy training.
In particular, note that the danger is not due to control or to generalized policy iteration.
Those cases are more complex to analyze, but the instability arises in the simpler prediction
case whenever it includes all three elements of the deadly triad. The danger is also not
due to learning or to uncertainties about the environment, because it occurs just as
strongly in planning methods, such as dynamic programming, in which the environment
is completely known.
If any two elements of the deadly triad are present, but not all three, then instability
can be avoided. It is natural, then, to go through the three and see if there is any one
that can be given up.
Of the three, function approximation most clearly cannot be given up. We needmethods that scale to large problems and to great expressive power. We need at least
linear function approximation with many features and parameters. State aggregation or
nonparametric methods whose complexity grows with data are too weak or too expensive.
Least-squares methods such as LSTD are of quadratic complexity and are therefore too
expensive for large problems.
Doing without bootstrapping is possible, at the cost of computational and data efficiency.
Perhaps most important are the losses in computational efficiency. Monte Carlo (nonbootstrapping) methods require memory to save everything that happens between making

11.3. The Deadly Triad265

each prediction and obtaining the final return, and all their computation is done once thefinal return is obtained. The cost of these computational issues is not apparent on serial
von Neumann computers, but would be on specialized hardware. With bootstrapping andeligibility traces (Chapter 12), data can be dealt with when and where it is generated,then need never be used again. The savings in communication and memory made possible
by bootstrapping are great.
The losses in data efficiency by giving up bootstrapping are also significant. We
have seen this repeatedly, such as in Chapters 7 (Figure 7.2) and 9 (Figure 9.2), wheresome degree of bootstrapping performed much better than Monte Carlo methods on
the random-walk prediction task, and in Chapter 10 where the same was seen on the
Mountain-Car control task (Figure 10.4). Many other problems show much faster learningwith bootstrapping (e.g., see Figure 12.14). Bootstrapping often results in faster learning
because it allows learning to take advantage of the state property, the ability to recognize
a state upon returning to it. On the other hand, bootstrapping can impair learning on
problems where the state representation is poor and causes poor generalization (e.g.,this seems to be the case on Tetris, see Şimşek, Algórta, and Kothiyal, 2016). A poor
state representation can also result in bias; this is the reason for the poorer bound on
the asymptotic approximation quality of bootstrapping methods (Equation 9.14). On
balance, the ability to bootstrap has to be considered extremely valuable. One may
sometimes choose not to use it by selecting long n-step updates (or a large bootstrapping
parameter, ⇡ 1; see Chapter 12) but often bootstrapping greatly increases efficiency. It
is an ability that we would very much like to keep in our toolkit.
Finally, there is o↵-policy learning; can we give that up? On-policy methods are often
adequate. For model-free reinforcement learning, one can simply use Sarsa rather thanQ-learning. O↵-policy methods free behavior from the target policy. This could beconsidered an appealing convenience but not a necessity. However, o↵-policy learning
is essential to other anticipated use cases, cases that we have not yet mentioned in thisbook but may be important to the larger goal of creating a powerful intelligent agent.
In these use cases, the agent learns not just a single value function and single policy,
but large numbers of them in parallel. There is extensive psychological evidence thatpeople and animals learn to predict many di↵erent sensory events, not just rewards. We
can be surprised by unusual events, and correct our predictions about them, even if
they are of neutral valence (neither good nor bad). This kind of prediction presumably
underlies predictive models of the world such as are used in planning. We predict what
we will see after eye movements, how long it will take to walk home, the probability ofmaking a jump shot in basketball, and the satisfaction we will get from taking on a new
project. In all these cases, the events we would like to predict depend on our acting ina certain way. To learn them all, in parallel, requires learning from the one stream ofexperience. There are many target policies, and thus the one behavior policy cannot
equal all of them. Yet parallel learning is conceptually possible because the behavior
policy may overlap in part with many of the target policies. To take full advantage ofthis requires o↵-policy learning.

266

11.4

Chapter 11: O↵-policy Methods with Approximation

Linear Value-function Geometry

To better understand the stability challenge of o↵-policy learning, it is helpful to think
about value function approximation more abstractly and independently of how learning
is done. We can imagine the space of all possible state-value functions—all functions
from states to real numbers v : S ! R. Most of these value functions do not correspondto any policy. More important for our purposes is that most are not representable by thefunction approximator, which by design has far fewer parameters than there are states.
Given an enumeration of the state space S = {s1 , s2 , . . . , s|S| }, any value function v
corresponds to a vector listing the value of each state in order [v(s1 ), v(s2 ), . . . , v(s|S| )]> .
This vector representation of a value function has as many components as there arestates. In most cases where we want to use function approximation, this would be far
too many components to represent the vector explicitly. Nevertheless, the idea of thisvector is conceptually useful. In the following, we treat a value function and its vector
representation interchangeably.
To develop intuitions, consider the case with three states S = {s1 , s2 , s3 } and two
parameters w = (w1 , w2 )> . We can then view all value functions/vectors as points ina three-dimensional space. The parameters provide an alternative coordinate systemover a two-dimensional subspace. Any weight vector w = (w1 , w2 )> is a point in thetwo-dimensional subspace and thus also a complete value function vw that assigns values
to all three states. With general function approximation the relationship between thefull space and the subspace of representable functions could be complex, but in the caseof linear value-function approximation the subspace is a simple plane, as suggested by
Figure 11.3.
Now consider a single fixed policy ⇡. We assume that its true value function, v⇡ , is toocomplex to be represented exactly as an approximation. Thus v⇡ is not in the subspace;in the figure it is depicted as being above the planar subspace of representable functions.
If v⇡ cannot be represented exactly, what representable value function is closest to
it? This turns out to be a subtle question with multiple answers. To begin, we needa measure of the distance between two value functions. Given two value functions v1and v2 , we can talk about the vector di↵erence between them, v = v1 v2 . If v is small,then the two value functions are close to each other. But how are we to measure the sizeof this di↵erence vector? The conventional Euclidean norm is not appropriate because,
as discussed in Section 9.2, some states are more important than others because theyoccur more frequently or because we are more interested in them (Section 9.11). As
in Section 9.2, let us use the distribution µ : S ! [0, 1] to specify the degree to which
we care about di↵erent states being accurately valued (often taken to be the on-policy
distribution). We can then define the distance between value functions using the normX
2 .
kvkµ =µ(s)v(s)2 .
(11.11)s2S

Note that the VE from Section 9.2 can be written simply using this norm as VE(w) =2
kvw v⇡ kµ . For any value function v, the operation of finding its closest value functionin the subspace of representable value functions is a projection operation. We define a

Be
llm
an

err
o

rv

ect
or

(B

E)

Value error (VE)Th99) extende form:it to least squares (Oits19projected
to as Bellman(1995,
referredstate:
nded
n
from
each
te
te
ex
of
3)
is
(1985). Baird
00
n
(2
io
r
at
v = ⇧B⇡ v ,or, and Mei
BE minimiz
e literature,
Engel, Mann
th
In
.
L
D
T
⇡ = argm
ss
ce
ro
P
n
Gaussia⇡
Unliketheoriginal
Bellmanequation,formostfunction
approximato
.
n
io
at
inimiz Value-function Geometry
267
dual mLinear
resi11.4.
:
the projected Bellman equation can be solved
exactly.
If
it
can’t
be
so
uationrd Bellman eq
wheree projecte
ellman errothe mean-squared
th
B
e
lv
ed
minimizeprojected
Bellman
error
: ⇥
so
ct
y
je
el
ximat
2.3 Pron is to appro
io
1)
at
(1
m
xi
ro
p
v
(s)
=
E⇡ Rt+1 +⇤ Rt+2
X
al for ap⇥ ⇡2
The third go.
v
d(s) n(⇧(B) the v ))(s) .
⇡v
esonv = ⇧B⇡ PBE(✓) =
r
ea
li
.,
.g
s (e
88,
approximatsorS 2 od[0,1)ttonis 19known
as the dis
ost function where
The 3D space ofmrmeth s (Su
fo
L
,
D
n
T
io
,
at
al
u
in
96
eq
ig
19
B
v
or
an
to
e
h
⇡
w
lm
ar
B expectation is condy. T
al Belindicates
that&theall orvaluefunctions
iginThesolved exactl
TDL (Bradkefixpoint,
minimumat
Unlike the overderwhichuaresprojection
sqthe
tation can be is achieveduto consiat
as3
states
eq
le
is
s
an
on
oe
d
lm
m
el
m
as
B
,
co
The
functionv
iscalledthe state-valu
ed
on
ss
ct
ti
le
⇡
je
lu
;
ro
so
p
this
is common⇥ D (e.g., Sutton et al. ⇤2009)
converge toexactlyX
)
1)
92
(1
19
g
n
an
vi
-T
ay
r
ie
D
ach
key
subproblem
underlying
adient A ⇡v )(s)rrov (s) = 0. almo
the erormvof(s)The goal ofy work on grd(s) e(Bd-weighted nive. The earlBoyan 1999).
th
ct
g
je
in
ob
iz
im
s
S
an
in
evaluation, the computation
g it asmin TDE or estima
y proposed mapproximatinror:have explicitl
er
to
t
an
rs
lm
fi
el
e
B
b
popularDP
algorithm
as policy
2)(1knownpppojectedappears toe call the pr
er
h
e
w
h
⇤
⇤
⇤
ic
(min
VE)
h
VE
BE
PBE
⇧v
=
v
v
v BE
wTD
⇡
of policies,
eachof whichisin (11), w⇧B⇡ va||.sequence
VE
PBE
v
,
v
at
g
PBE(✓) = ||
in
rt
ta
S
1.
re
PBE
0
Figu=
TDL,
algorithms
⇧Bat
⇡ vth
we lefound.
ft side of In
takes us such as TD
vw
ing
n operator
ok
io
lo
ct
je
by
ro
min
p
d
e
oo
st
Th e
the
for
as partd thcurrent policy,.example
BEBE
P
bspace, an
is best unofder
ive subspace
the sualluvaluefunctionsrepresentable
ashverwe we started is theeid
wh1is ob jectThe
ts
ou
s
T
w
s
d
ke
an
ta
.
p
ce
u
or
d
pla
If thethstateis finite, then th
operate same spacen where we en
the wBellman
k leaves us inactance betwee
b
is
d
d
2
e
an
h
T
p
u
.
it
computer as a large array with one ent
back into
when the trip
inimal (zero)
distance is mform the estimate.
Such tabular meth4

Figure 11.3: The geometry of linear value-function
approximation. Shown is the three8
throughdiscretization,
aggr
dimensional space of all value functions over three states,ones,
while shown
as a planeis the subspace stateof>
all value functions representable by a linear function approximatorwithparameter
w
=
(w
,
w
)
.
1
2 these meof the state space increases,
The true value function v⇡ is in the larger space and can be projected down (into the subspace,ine↵ective. This is the e↵ect which gav
using a projection operator ⇧) to its best approximation in the value error (VE) sense. The
best approximators in the Bellman error (BE), projected Bellman
error(PBE), andand temporal
A moregeneralflexible approadi↵erence error (TDE) senses are all potentially di↵erent and are shown in the lower right. (VE,
form of fixed size and fixed structure wi
BE, and PBE are all treated as the corresponding vectors in this figure.) The Bellman operator
are canthenchangedto reshape
the approx
takes a value function in the plane to one outside, which
thenbe projectedback. If you
iteratively applied the Bellman operator outside the space(shownWe
in gray
above)the
youparameterized
wouldfunction.
denotereach the true value function, as in conventional dynamic programming. If instead you kept
projecting back into the subspace at each step, as in the lower step shown in gray, then the fixed
point would be the point of vector-zero PBE.
v (s) ⇡

, with n function⌧ |S|, is the
projection operator ⇧ that takes an arbitrary value where
function✓to 2
theRrepresentablethat is closest in our norm:
function can have arbitrary form as lon

the weights. For example, it could be(11.12)layer neural network where ✓ is the conrefer to ✓ exclusively as the weights, o
The representable value function that is closest to the true value function v⇡ is thus its
thingslikeasymptoticallythe discount-rate
projection, ⇧v⇡ , as suggested in Figure 11.3. Thisfor
is thesolutionfound paramby Monte Carlo methods, albeit often very slowly. TheAn
projection
operationisdiscussed
important special case is that imore fully in the box on the next page.
the weights and in features of the state.
⇧v = vw where w = argmin kv
w2Rd2vw kµ .

TD methods find di↵erent solutions. To understand their rationale, recall that the
Bellman equation for value function v⇡ isXXv⇡ (s) =⇡(a|s)p(s0 , r |s, a) [r + v⇡ (s0 )] ,for all s 2 S.
(11.16)
a

s0 ,r

v

where the (s) 2 Rn , s 2 S, are fe
denotes the inner product of two vecto

268

Chapter 11: O↵-policy Methods with Approximation

The projection matrix
For a linear function approximator, the projection operation is linear, which implies
that it can be represented as an |S| ⇥ |S| matrix:.
⇧ = X X> DX

1

X> D,

(11.13)where, as in Section 9.4, D denotes the |S| ⇥ |S| diagonal matrix with the µ(s)on the diagonal, and X denotes the |S| ⇥ d matrix whose rows are the feature
vectors x(s)> , one for each state s. If the inverse in (11.13) does not exist, then thepseudoinverse is substituted. Using these matrices, the squared norm of a vector
can be written
2kvkµ = v > Dv,(11.14)and the approximate linear value function can be written
vw = Xw.

(11.15)The true value function v⇡ is the only value function that solves (11.16) exactly. If an
approximate value function vw were substituted for v⇡ , the di↵erence between the rightand left sides of the modified equation could be used as a measure of how far o↵ vw isfrom v⇡ . We call this the Bellman error at state s:
0
1
X
X
.
0
0
¯w (s) = @⇡(a|s)p(s , r |s, a) [r + vw (s )]A
vw (s)(11.17)
a

⇥

s0 ,r

= E⇡ Rt+1 + vw (St+1 )vw (St )⇤St = s, At ⇠ ⇡ ,(11.18)which shows clearly the relationship of the Bellman error to the TD error (11.3). The
Bellman error is the expectation of the TD error.
The vector of all the Bellman errors, at all states, ¯w 2 R|S| , is called the Bellman
error vector (shown as BE in Figure 11.3). The overall size of this vector, in the norm, isan overall measure of the error in the value function, called the mean square Bellman
error :
2
BE(w) = ¯w µ .

(11.19)It is not possible in general to reduce the BE to zero (at which point vw = v⇡ ), but for
linear function approximation there is a unique value of w for which the BE is minimized.
This point in the representable-function subspace (labeled min BE in Figure 11.3) isdi↵erent in general from that which minimizes the VE (shown as ⇧v⇡ ). Methods that
seek to minimize the BE are discussed in the next two sections.
The Bellman error vector is shown in Figure 11.3 as the result of applying the Bellman
operator B⇡ : R|S| ! R|S| to the approximate value function. The Bellman operator is11.5. Gradient Descent in the Bellman Error

269

defined by
X. X(B⇡ v)(s) =⇡(a|s)p(s0 , r |s, a) [r + v(s0 )] ,(11.20)s0 ,ra

for all s 2 S, v : S ! R. The Bellman error vector for vw can be written ¯w = B⇡ vw vw .
If the Bellman operator is applied to a value function in the representable subspace,then, in general, it will produce a new value function that is outside the subspace, assuggested in the figure. In dynamic programming (without function approximation), thisoperator is applied repeatedly to the points outside the representable space, as suggestedby the gray arrows in the top of Figure 11.3. Eventually that process converges to thetrue value function v⇡ , the only fixed point for the Bellman operator, the only valuefunction for whichv ⇡ = B⇡ v ⇡ ,(11.21)

which is just another way of writing the Bellman equation for ⇡ (11.16).
With function approximation, however, the intermediate value functions lying outside
the subspace cannot be represented. The gray arrows in the upper part of Figure 11.3
cannot be followed because after the first update (dark line) the value function must
be projected back into something representable. The next iteration then begins within
the subspace; the value function is again taken outside of the subspace by the Bellman
operator and then mapped back by the projection operator, as suggested by the lower
gray arrow and line. Following these arrows is a DP-like process with approximation.
In this case we are interested in the projection of the Bellman error vector back into
the representable space. This is the projected Bellman error vector ⇧ ¯w , shown in
Figure 11.3 as PBE. The size of this vector, in the norm, is another measure of error in
the approximate value function. For any approximate value function vw , we define themean square Projected Bellman error, denoted PBE, as
2

PBE(w) = ⇧ ¯w µ .

(11.22)

With linear function approximation there always exists an approximate value function(within the subspace) with zero PBE; this is the TD fixed point, wTD , introduced in
Section 9.4. As we have seen, this point is not always stable under semi-gradient TD
methods and o↵-policy training. As shown in the figure, this value function is generallydi↵erent from those minimizing VE or BE. Methods that are guaranteed to converge toit are discussed in Sections 11.7 and 11.8.

11.5

Gradient Descent in the Bellman Error

Armed with a better understanding of value function approximation and its various
objectives, we return now to the challenge of stability in o↵-policy learning. We would
like to apply the approach of stochastic gradient descent (SGD, Section 9.3), in which
updates are made that in expectation are equal to the negative gradient of an objective

270

Chapter 11: O↵-policy Methods with Approximation

function. These methods always go downhill (in expectation) in the objective and becauseof this are typically stable with excellent convergence properties. Among the algorithms
investigated so far in this book, only the Monte Carlo methods are true SGD methods.
These methods converge robustly under both on-policy and o↵-policy training as well
as for general nonlinear (di↵erentiable) function approximators, though they are often
slower than semi-gradient methods with bootstrapping, which are not SGD methods.
Semi-gradient methods may diverge under o↵-policy training, as we have seen earlier inthis chapter, and under contrived cases of nonlinear function approximation (Tsitsiklis
and Van Roy, 1997). With a true SGD method such divergence would not be possible.
The appeal of SGD is so strong that great e↵ort has gone into finding a practicalway of harnessing it for reinforcement learning. The starting place of all such e↵orts isthe choice of an error or objective function to optimize. In this and the next section
we explore the origins and limits of the most popular proposed objective function, thatbased on the Bellman error introduced in the previous section. Although this has been a
popular and influential approach, the conclusion that we reach here is that it is a misstep
and yields no good learning algorithms. On the other hand, this approach fails in aninteresting way that o↵ers insight into what might constitute a good approach.
To begin, let us consider not the Bellman error, but something more immediateand naive. Temporal di↵erence learning is driven by the TD error. Why not take theminimization of the expected square of the TD error as the objective? In the generalfunction-approximation case, the one-step TD error with discounting is
t = Rt+1 +v̂(St+1 ,wt )v̂(St ,wt ).

A possible objective function then is what one might call the mean square TD error :X⇥⇤
TDE(w) =µ(s)E t2 St = s, At ⇠ ⇡
s2S

=

X
s2S

⇥
µ(s)E ⇢t t2

St = s, At ⇠ b⇥⇤= Eb ⇢t t2 .

⇤(if µ is the distribution encountered under b)

The last equation is of the form needed for SGD; it gives the objective as an expectationthat can be sampled from experience (remember the experience is due to the behavior
policy b). Thus, following the standard SGD approach, one can derive the per-step updatebased on a sample of this expected value:
wt+1 = wt= wt

1↵r(⇢t t2 )2↵⇢t t r t

= wt + ↵⇢t t rv̂(St ,wt )rv̂(St+1 ,wt ) ,(11.23)

which you will recognize as the same as the semi-gradient TD algorithm (11.2) except for
the additional final term. This term completes the gradient and makes this a true SGD
algorithm with excellent convergence guarantees. Let us call this algorithm the naive

11.5. Gradient Descent in the Bellman Error

271

residual-gradient algorithm (after Baird, 1995). Although the naive residual-gradientalgorithm converges robustly, it does not necessarily converge to a desirable place.
Example 11.2: A-split example,
showing the naiveté of the naive residual-gradient algorithm
Consider the three-state episodic MRP shown to the right.
Episodes begin in state A and then ‘split’ stochastically, half
1
B
the time going to B (and then invariably going on to terminate0with a reward of 1) and half the time going to state C (andthen invariably terminating with a reward of zero). Reward for A
0
the first transition, out of A, is always zero whichever way the
C 0
episode goes. As this is an episodic problem, we can take tobe 1. We also assume on-policy training, so that ⇢t is always
1, and tabular function approximation, so that the learning algorithms are free togive arbitrary, independent values to all three states. Thus, this should be an easy
problem.
What should the values be? From A, half the time the return is 1, and half thetime the return is 0; A should have value 12 . From B the return is always 1, so its
value should be 1, and similarly from C the return is always 0, so its value shouldbe 0. These are the true values and, as this is a tabular problem, all the methods
presented previously converge to them exactly.
However, the naive residual-gradient algorithm finds di↵erent values for B and
C. It converges with B having a value of 34 and C having a value of 14 (A converges
correctly to 12 ). These are in fact the values that minimize the TDE.
Let us compute the TDE for these values. The first transition of each episode iseither up from A’s 12 to B’s 34 , a change of 14 , or down from A’s 12 to C’s 14 , a changeof 14 . Because the reward is zero on these transitions, and = 1, these changes are1
the TD errors, and thus the squared TD error is always 16
on the first transition.
3
The second transition is similar; it is either up from B’s 4 to a reward of 1 (and aterminal state of value 0), or down from C’s 14 to a reward of 0 (again with a terminal
1
state of value 0). Thus, the TD error is always ± 14 , for a squared error of 16
on the1second step. Thus, for this set of values, the TDE on both steps is 16 .
Now let’s compute the TDE for the true values (B at 1, C at 0, and A at 12 ). In thiscase the first transition is either from 12 up to 1, at B, or from 12 down to 0, at C; ineither case the absolute error is 12 and the squared error is 14 . The second transition
has zero error because the starting value, either 1 or 0 depending on whether thetransition is from B or C, always exactly matches the immediate reward and return.
Thus the squared TD error is 14 on the first transition and 0 on the second, for a1mean reward over the two transitions of 18 . As 18 is bigger that 16, this solution isworse according to the TDE. On this simple problem, the true values do not havethe smallest TDE.

272

Chapter 11: O↵-policy Methods with Approximation

A tabular representation is used in the A-split example, so the true state values canbe exactly represented, yet the naive residual-gradient algorithm finds di↵erent values,
and these values have lower TDE than do the true values. Minimizing the TDE is naive;by penalizing all TD errors it achieves something more like temporal smoothing thanaccurate prediction.
A better idea would seem to be minimizing the mean square Bellman error (BE). If
the exact values are learned, the Bellman error is zero everywhere. Thus, a Bellmanerror-minimizing algorithm should have no trouble with the A-split example. We cannot
expect to achieve zero Bellman error in general, as it would involve finding the true valuefunction, which we presume is outside the space of representable value functions. But
getting close to this ideal is a natural-seeming goal. As we have seen, the Bellman error
is also closely related to the TD error. The Bellman error for a state is the expected TD
error in that state. So let’s repeat the derivation above with the expected TD error (allexpectations here are implicitly conditional on St ):1
2↵r(E⇡[ t ] )
2
1
2
= wt
↵r(Eb[⇢t t ] )2= wt ↵Eb[⇢t t ] rEb[⇢t t ]⇥⇤= wt ↵Eb ⇢t (Rt+1 + v̂(St+1 ,w) v̂(St ,w)) Eb[⇢t r t ]h ⇥ih⇤= wt + ↵ Eb ⇢t (Rt+1 + v̂(St+1 ,w))v̂(St ,w) rv̂(St ,w)

wt+1 = wt⇥⇤i
Eb ⇢t rv̂(St+1 ,w) .

This update and various ways of sampling it are referred to as the residual-gradientalgorithm. If you simply used the sample values in all the expectations, then the equation
above reduces almost exactly to (11.23), the naive residual-gradient algorithm.1 But
this is naive, because the equation above involves the next state, St+1 , appearing in two
expectations that are multiplied together. To get an unbiased sample of the product,
two independent samples of the next state are required, but during normal interaction
with an external environment only one is obtained. One expectation or the other can be
sampled, but not both.
There are two ways to make the residual-gradient algorithm work. One is in the caseof deterministic environments. If the transition to the next state is deterministic, then
the two samples will necessarily be the same, and the naive algorithm is valid. The
other way is to obtain two independent samples of the next state, St+1 , from St , one forthe first expectation and another for the second expectation. In real interaction withan environment, this would not seem possible, but when interacting with a simulated
environment, it is. One simply rolls back to the previous state and obtains an alternate
next state before proceeding forward from the first next state. In either of these cases theresidual-gradient algorithm is guaranteed to converge to a minimum of the BE under theusual conditions on the step-size parameter. As a true SGD method, this convergence is
1 For state values there remains a small di↵erence in the treatment of the importance sampling ratio⇢t . In the analagous action-value case (which is the most important case for control algorithms), theresidual-gradient algorithm would reduce exactly to the naive version.

11.5. Gradient Descent in the Bellman Error

273

robust, applying to both linear and nonlinear function approximators. In the linear case,convergence is always to the unique w that minimizes the BE.
However, there remain at least three ways in which the convergence of the residualgradient method is unsatisfactory. The first of these is that empirically it is slow, muchslower that semi-gradient methods. Indeed, proponents of this method have proposed
increasing its speed by combining it with faster semi-gradient methods initially, thengradually switching over to residual gradient for the convergence guarantee (Baird andMoore, 1999). The second way in which the residual-gradient algorithm is unsatisfactoryis that it still seems to converge to the wrong values. It does get the right values in all
tabular cases, such as the A-split example, as for those an exact solution to the Bellman

Example 11.3: A-presplit example, a counterexample for the BE
Consider the three-state episodic MRP shown to theright: Episodes start in either A1 or A2, with equal
10A1B
probability. These two states look exactly the same tothe function approximator, like a single state A whoseAfeature representation is distinct from and unrelated to
the feature representation of the other two states, B andA2
C 0
0C, which are also distinct from each other. Specifically,
the parameter of the function approximator has threecomponents, one giving the value of state B, one giving the value of state C, and one
giving the value of both states A1 and A2. Other than the selection of the initialstate, the system is deterministic. If it starts in A1, then it transitions to B with areward of 0 and then on to termination with a reward of 1. If it starts in A2, then it
transitions to C, and then to termination, with both rewards zero.
To a learning algorithm, seeing only the features, the system looks identical to
the A-split example. The system seems to always start in A, followed by either
B or C with equal probability, and then terminating with a 1 or a 0 depending
deterministically on the previous state. As in the A-split example, the true valuesof B and C are 1 and 0, and the best shared value of A1 and A2 is 12 , by symmetry.
Because this problem appears externally identical to the A-split example, we
already know what values will be found by the algorithms. Semi-gradient TD
converges to the ideal values just mentioned, while the naive residual-gradientalgorithm converges to values of 34 and 14 for B and C respectively. All statetransitions are deterministic, so the non-naive residual-gradient algorithm will also
converge to these values (it is the same algorithm in this case). It follows then that
this ‘naive’ solution must also be the one that minimizes the BE, and so it is. On a
deterministic problem, the Bellman errors and TD errors are all the same, so the
BE is always the same as the TDE. Optimizing the BE on this example gives rise to
the same failure mode as with the naive residual-gradient algorithm on the A-split
example.

274

Chapter 11: O↵-policy Methods with Approximation

equation is possible. But if we examine examples with genuine function approximation,then the residual-gradient algorithm, and indeed the BE objective, seem to find thewrong value functions. One of the most telling such examples is the variation on the
A-split example known as the A-presplit example, shown on the preceding page, in which
the residual-gradient algorithm finds the same poor solution as its naive version. This
example shows intuitively that minimizing the BE (which the residual-gradient algorithm
surely does) may not be a desirable goal.
The third way in which the convergence of the residual-gradient algorithm is notsatisfactory is explained in the next section. Like the second way, the third way is also
a problem with the BE objective itself rather than with any particular algorithm forachieving it.

11.6

The Bellman Error is Not Learnable

The concept of learnability that we introduce in this section is di↵erent from that
commonly used in machine learning. There, a hypothesis is said to be “learnable” if
it is efficiently learnable, meaning that it can be learned within a polynomial rather
than an exponential number of examples. Here we use the term in a more basic way,
to mean learnable at all, with any amount of experience. It turns out many quantities
of apparent interest in reinforcement learning cannot be learned even from an infinite
amount of experiential data. These quantities are well defined and can be computed
given knowledge of the internal structure of the environment, but cannot be computedor estimated from the observed sequence of feature vectors, actions, and rewards.2 We
say that they are not learnable. It will turn out that the Bellman error objective (BE)introduced in the last two sections is not learnable in this sense. That the Bellman error
objective cannot be learned from the observable data is probably the strongest reason
not to seek it.
To make the concept of learnability clear, let’s start with some simple examples.
Consider the two Markov reward processes3 (MRPs) diagrammed below:

0
0

w

2

0

w

w

2

2
Where two edges leave a state, both transitions are assumed to occur with equal probability,
and the numbers indicate the reward received. All the states appear the same; they all
produce the same single-component feature vector x = 1 and have approximated value
w. Thus, the only varying part of the data trajectory is the reward sequence. The left
MRP stays in the same state and emits an endless stream of 0s and 2s at random, eachwith 0.5 probability. The right MRP, on every step, either stays in its current state or
2 They would of course be estimated if the state sequence were observed rather than only thecorresponding feature vectors.
3 All MRPs can be considered MDPs with a single action in all states; what we conclude about MRPs
here applies as well to MDPs.

11.6. The Bellman Error is Not Learnable275

switches to the other, with equal probability. The reward is deterministic in this MRP,
always a 0 from one state and always a 2 from the other, but because the each stateis equally likely on each step, the observable data is again an endless stream of 0s and2s at random, identical to that produced by the left MRP. (We can assume the right
MRP starts in one of two states at random with equal probability.) Thus, even given
an infinite amount of data, it would not be possible to tell which of these two MRPs
was generating it. In particular, we could not tell if the MRP has one state or two, isstochastic or deterministic. These things are not learnable.
This pair of MRPs also illustrates that the VE objective (9.1) is not learnable. If= 0, then the true values of the three states (in both MRPs), left to right, are 1, 0,
and 2. Suppose w = 1. Then the VE is 0 for the left MRP and 1 for the right MRP.
Because the VE is di↵erent in the two problems, yet the data generated has the same
distribution, the VE cannot be learned. The VE is not a unique function of the data
distribution. And if it cannot be learned, then how could the VE possibly be useful asan objective for learning?
If an objective cannot be learned, it does indeed draw its utility into question. In
the case of the VE, however, there is a way out. Note that the same solution, w = 1,
is optimal for both MRPs above (assuming µ is the same for the two indistinguishable
states in the right MRP). Is this a coincidence, or could it be generally true that all
MDPs with the same data distribution also have the same optimal parameter vector? If
this is true—and we will show next that it is—then the VE remains a usable objective.
The VE is not learnable, but the parameter that optimizes it is!
To understand this, it is useful to bring in another natural objective function, thistime one that is clearly learnable. One error that is always observable is that between thevalue estimate at each time and the return from that time. The mean square return error,
denoted RE, is the expectation, under µ, of the square of this error. In the on-policy case
the RE can be writtenhi2
RE(w) = E Gt v̂(St ,w)hi2= VE(w) + E Gt v⇡ (St ).
(11.24)Thus, the two objectives are the same except for a variance term that does not depend on
the parameter vector. The two objectives must therefore have the same optimal parametervalue w⇤ . The overall relationships are summarized in the left side of Figure 11.4.
⇤

Exercise 11.4 Prove (11.24). Hint: Write the RE as an expectation over possible statess of the expectation of the squared error given that St = s. Then add and subtract thetrue value of state s from the error (before squaring), grouping the subtracted true value
with the return and the added true value with the estimated value. Then, if you expand
the square, the most complex term will end up being zero, leaving you with (11.24). ⇤Now let us return to the BE. The BE is like the VE in that it can be computed fromknowledge of the MDP but is not learnable from data. But it is not like the VE in that its
minimum solution is not learnable. The box on the next page presents a counterexample—
two MRPs that generate the same data distribution but whose minimizing parameter
vector is di↵erent, proving that the optimal parameter vector is not a function of the276

Chapter 11: O↵-policy Methods with Approximation

Example 11.4: Counterexample to the learnability of the Bellman error
To show the full range of possibilities we need a slightly more complex pair of Markov
reward processes (MRPs) than those considered earlier. Consider the following two
MRPs:
-1
1
1
-1
-1 

A

B

0

A

B

B

0
0

Where two edges leave a state, both transitions are assumed to occur with equal
probability, and the numbers indicate the reward received. The MRP on the left has
two states that are represented distinctly. The MRP on the right has three states,
two of which, B and B0 , appear the same and must be given the same approximate
value. Specifically, w has two components and the value of state A is given by the firstcomponent and the value of B and B0 is given by the second. The second MRP has
been designed so that equal time is spent in all three states, so we can take µ(s) = 13 ,for all s.
Note that the observable data distribution is identical for the two MRPs. In bothcases the agent will see single occurrences of A followed by a 0, then some number
of apparent Bs, each followed by a 1 except the last, which is followed by a 1, thenwe start all over again with a single A and a 0, etc. All the statistical details are thesame as well; in both MRPs, the probability of a string of k Bs is 2 k .
Now suppose w = 0. In the first MRP, this is an exact solution, and the BE iszero. In the second MRP, this solution produces a squared error in both B and B0 of
1, such that BE = µ(B)1 + µ(B0 )1 = 23 . These two MRPs, which generate the samedata distribution, have di↵erent BEs; the BE is not learnable.
Moreover (and unlike the earlier example for the VE) the minimizing value of w
is di↵erent for the two MRPs. For the first MRP, w = 0 minimizes the BE for any. For the second MRP, the minimizing w is a complicated function of , but inthe limit, as ! 1, it is ( 12 , 0)> . Thus the solution that minimizes BE cannot be
estimated from data alone; knowledge of the MRP beyond what is revealed in thedata is required. In this sense, it is impossible in principle to pursue the BE as an
objective for learning.
It may be surprising that in the second MRP the BE-minimizing value of A is so far
from zero. Recall that A has a dedicated weight and thus its value is unconstrainedby function approximation. A is followed by a reward of 0 and transition to a statewith a value of nearly 0, which suggests vw (A) should be 0; why is its optimalvalue substantially negative rather than 0? The answer is that making vw (A) negative
reduces the error upon arriving in A from B. The reward on this deterministic transitionis 1, which implies that B should have a value 1 more than A. Because B’s value isapproximately zero, A’s value is driven toward 1. The BE-minimizing value of ⇡ 12
for A is a compromise between reducing the errors on leaving and on entering A.

11.6. The Bellman Error is Not Learnable

Monte Carlo
objectives

277

policy together completely determine the probability di

policy together completely determine the proba
Bootstrapping
Assumeforthe moment
thatthetheprobability
state,
andData policy together
Data
policy completely
togethercompletely
determine
distributio
Assumeforthemomentthat
theaction,state,
actiondeterminetheprobability
distribution
over
dare
objectives
distribution
distribution
,
a
,
r
,
.
.
.
,
r
,
,
there
for
any
finitesequence
=
0
0
1
,
a
,
r
,
.
.
.
,
r
,
for
any
finitesequence=
Assumefor the thatmomentthat thestate,andaction,seforthe moment
the state,action,rewardarek all0 and
0 k sets
1 kreward
kis,mine the probability distribution over Assumedata trajectories.

sibly sequencezero)
of it, r=occuring
portiona prtrinitialras.there. , rinitial, iskthe,a there
is ofaportion
wellforThen,any
finiterit0k, occuring
welldefinedfinitesequence= sibly0. ,, a1k,,.thek as
0 , a0zero)
1 , . .of
he state, action, and reward sets are for
all any
finite.
)
=
,
A
=
a
,
R
=
r
,
.
.
.
,
R
=
r
,
P(
)
=
Pr{
(S
)
A
a10ofRa1 =rk1 , . . .k,we
R
P(the
) =
Pr{the0 as0(Sinitialof it asoccuring
trajectory,zero)siblyof(positzero)occuring
initialportionof0portiona 01trajectory,which
00
probability
0 , r1 , . . . , rk , k , there is a well defined sibly
then
is
a
completecharacterizationofasource
of
data
tr
then
is
a
completecharacterizationofasource
o
)
=
,
A
=
a
,
R
=
r
,
.
.
.
,
R
=
r
,
(S
)
=
P(
)
=
Pr{
(S
)
=
,
A
=
a
,
R
=
r
,
.
.
.
,
R
=
r
,
(S
)
=
}.
The
P(
)
=
Pr{
(S
0
0
0
0
1
1
k
k
k
0
0
0
1
1
k
k
k
k
MDP
initial portion of aMDP
trajectory,which
1
2 we may denoted
MDP1 everything
MDP
everything
aboutoftheof thedata,
but
itToisknow
stil
about
the
statistics
of
the
data,
but
2statistics
then
is
a
completecharacterizationofasourceofdata
trajectorie
then
is
a
completecharacterizationasourceofdata
trajectories.
R1 = r1 , . . . , Rk = rk , (Sk ) = k }. The distribution P
particular,theVEobjectives
areparticular,
theofBE
VE
andobjectives
arecomput
readilyRE
everythingthestatistics
the
data,
but
it readily
isthan
still
less
tha
everythingstatistics
ofthe and
data,
but
it BE
is stilllessknowinn of a source of data trajectories. To know
P isMSRE
toabout
knowtheabout
MSTDE
PBE
TDE
MSPBE
Section
3,
but
these
cannotbedetermined
from
P MDP
alone.
Section3,objectives
butreadily
these arecannotbe determinedfromparticular,
the
VE
and
BE
readily
computedfromparticular,
the
VE
and
BE
objectives
arecomputedfromthethe data, but it is still less than knowing the MDP. In
VE1 1
VE2 2 Section 3, Section
MSVE
MSVE
these cannot
be determinedfrom P alone.
but thesecannotP alone.
ives are readily computed from the MDP
as described
in3, but
MSBE
MSBE
BE1 1be determined
BE2 2 from✓1 ✓2 ✓3✓1✓4✓2BE✓31 ✓BE
2 MDP
4 BE
1 BE12
termined from P alone.
⇤⇤✓4✓22BE✓1 ✓2 ✓3 ✓✓14 ✓BE
MDP
w
✓MDP
✓BE
✓313✓1 ✓BE
MDP
2 ✓13✓1BE
2MDP
212
244 1BE
1 BE
2 MDP
31 w
41✓2 BE⇤⇤⇤✓1RE✓3 ✓RE✓1 4 w
MDP✓BE✓4 2 BEMDP
BE1 BE2 MDP1 MDP2 PBE w✓
TDEw
1 2MDP
2 1 MDP2
1 BE
1 ✓TDE2 ✓13 BE
2 VE
2VE
The
problem
can
be
seen
in
very
simple,POM
The
problem
can
be
seen
in
very
simple,POMDP-like
TDE RE TDE
VE RE VE
✓4 BE1 BE2 MDP1 MDP2
Figure 11.4: Causal relationships among thedataproduced
distribution,
MDPs,
and
variousobjectives.
data
producedbytwodi↵erent
MDPs
isidentica
data
by
two
di↵erent
MDPs
isidentical
inever
The problem
be seenin bevery
simple,POMDP-likeexamples, example
in which
Thecanproblemcanseen
in very
simple, POMDP-like
Left, Monte Carlo objectives: Two di↵erent
MDPs
canproduce
same
distribution
Incasesucha BE
casethetheBEdatais notliterally
not a function
In
such
a
the
is
literallyafunctionofthedata producedby twois identicaleveryrespect,
the dBdata that
produced
twoMDPsdi↵erent
MDPs
isinidenticalin everyyet
respecyet also
produce di↵erent
the di↵erent
VEbyobjective
cannotbedata.
determinedry simple, POMDP-likeexamples,in whichVEs,
the proving
observableestimate
it fromOne
offromthe data
simplestexam
estimate
it
from
data.
One
of
the
simplest
examplesisandth⇤ data,In suchalla suchcase
theamust
BE
literallya function
thevector,
data,
and
In
such
caseishavethe
BE sameisnot
literallynotaoffunctionof w
theis notlearnable.
VEs
theoptimalparameter! thus therDPs is identical inand
every
respect,
yetHowever,
the
BE
is
di↵erent.
itestimate
from data.
Oneof thesimplest
examplesisthe pairisofthe
MDPs
it fromdata.
Oneof thesimplest
examplespair shofMoreover, this same w⇤estimate
can be determinedfrom
another
objective,
theRE, which1-1-1 is uniquely
not a function of determinedthe data, fromand thethusdata
there
is no way to1
distribution. Thus w⇤ and the RE are learnable even though the VEs
1
1 A
simplest examplesare
is not.
the pairof Bootstrapping
MDPs shown below:
AB can produce
B
Right,
objectives:Twodi↵erent
the same data
1
-1
1 A
-1MDPs
A vectors;A
B -1
distribution
and have di↵erent minimizing
parameter0-1 BEs1 yet also produce di↵erent
0
A
B
A
B
these are not learnablefrom the data distribution. The PBE and TDE objectives and their00
A minima
B can-1
B determined(di↵erent)
be directly
fromand thus are learnable.
0
0 data
These
MDPs
have
only
one
action(or,equivalent
These MDPs have only one action (or, equivalently,
0 no act0
chains.two
Where
edges
leave botha state,both poschains. Whereedgestwoleave
a state,possibilities
These MDPs
have
only haveone action(or,action
equivalently,
no on
actions),they
aresoth
in
These
MDPs
only
one
(or,
equivalently,
nosoactions),
probability.
The
numberstheedges
indicate
probability.
The
numbers onobjectives
the edges indicate
reward0data and thus cannot
be learned
fromit.
The twootherbootstrapping
thatassumed
we thechains.
Where
two
edges
leave
a
state,
both
possibilities
are
to
oc
chains.
Where
edges
leave
a
state,
both
possibilities
areassu
on hasthe twoleft states
has twostatesare reThe
MDPTheon MDP
the
left
that
are that
representehaveconsidered,thearePBE
and TDE,beThedetermined
from
data
(are
learnable)
and(or, equivalently, no
actions),so theyin e↵ect
Markovprobability.
Thecannumbersonthe edges
indicatetherewardemittedif emitted
thatTheedprobability.
numbers
on thattheedges
indicatetherewardweight
so
they
can
take
on
any
value.
weightsothat theycaneachtakeotheron anyThe MDP odetermine
optimal solutions
thatin leftgeneraldi↵erentfromandvalue.
theBEstate, both possibilities
are assumed
toThe
occur
with
equal
MDP
onare
the
statesthatarerepresented
distinctly;
each
The
MDP
onhas
thetwolefthas
two
statesthatarerepresented
distincand
B , arerepresented
identicallyof which,
B which,andsideBB, ofarerepresented
identicallyand musminimums.
Theifgeneralcaseisso
istraversed.
summarizedin takethe ofright
Figure11.4.
dges indicate the reward
emittedthatweightedge
that theycanonvalue.
The
MDP
on
the
right
has
weightso thattheycananytakeon imagine
anyvalue.
The
MDP
on
the thriAvalue.
We
can
that
the
value
of isstatevalue.
Wecanimagine
that
the
value
of
state
A
given
Thus, distinctly;
the BE is notlearnable;itcannotbeestimatedfromfeaturevectors
andotherates that are represented
each
has
a
separateof which, of
B and
B , Barerepresented
identicallyandmust andbe the
given
the
sam
which,
andB,
arerepresented
identically
must
be
give
is
given
by
second.
N
the
value
of
B
and
B
is given
by no
thealgorithm
second. Notice tha
the value ofsettings.
B and BThereobservable
limitsBEmodel-based
canbe
any value. The MDP
on thedata.
rightThis
hasvalue.
three thestates,
two
We
canto
imaginethatthethevalue
ofMDPs.
state
AIn
is
given
byis thefirst
compvalue.
Weforcanimagine
that
the
value
of
state
A
given
by
the
for
two
both
cases
the
agent
will
the
two MDPs.
Instates
bothbeyond
cases thewill see singminimizes
without
accessto theunderlying
MDP
theagent
featureed identically andthat
must
be giventhe
theBE
same
approximate
isthesecond.
theobservable
d
the
value
of BvalueandB
isnumbergiven
by
theNoticesecond.
Noticethat thetheof then
B given
and
0,Bby
then
some
numberof followed
Bsthateach
followedby aob
0,
some
of
Bs
each
by
algorithm
is only able to minimize BE because it is alloweda 1, excep
value of state A is vectors.
given byThe
theresidual-gradientfirst componentof✓the
andtwofor the two
InMDPs.
both 1,
cases
the
agent
will
see single
occurrences
ofA A
forMDPs.
In
both
cases
the
agent
will
see
singleoccurr
then
we
start
all
over
again
with
a
single1, thenwe starta single
A and a 0, ato double sample from the same state—not
a statethat allhasover
the again
same withfeaturevector,thenissome
numberofnumberBs eachfollowedby
a 1, theexceptthelast ofwhichisthe second. Notice that the observable0,data
identical
0, then
some
of
Bs
each
followedbya1,
except
as
well;
in
both
MDPs,
probability
a strin
well; in both
MDPs,
the seeprobability
a string theofklaB
but one that is guaranteed to be the sameasunderlying
state.
We cannow thatofthere
1, Athenwe1,start
over
again
with
a
single
A
and
a
0,
etc.
All
the
detailthe agent will see single occurrences offollowedby all
awe
=
0.
In
the
first
MDP,
this
is
an
exa
function
v
then
start
all
over
again
with
a
single
A
and
a
0,
etc.
All
0. Insuchthe first
MDP,
thisnominal,
is an exact
solutiofunctionv =someis no way around this. Minimizing the BE
requiresaccessto thek.
as well; inasboth
theMDPs,
probability
ofsolution
a string
ofakstring
Bs producesisanof
2 errorNowthesecondMDP,
thisp
solutionanerror
followed by a 1, underlying
except theMDP.
last which
by MDPs,
ainthewell;boththethisprobabilityofkk . Bs
is both2conpsecondMDP,
producesinThis isisanfollowed
importantlimitation
of
the
BE
beyondthat
identified
inthepan),exact
pfirst of
=
0.
In
the
MDP,
this
is
solution,andtheoverallfunction
v
d(B)
+
d(B
or
2/3
if
the
three
statesar= 0.
In+the
first
MDP,
is
exact
solution,and tfunction
ith a single A andA-presplit
a 0, etc. example
All the details
the
d(B)
d(B
), or
2/3thisif thethreestatesare equallyon pageare
273.
Allsamethis vof
directs
more
attention
toward
thean
PBE.
which
generate
the
same
data,
di↵erent
1,BE
forB
thepsecond
MDP,
this
solution
producesanerror
inanboth
Bhave
and
B of
the
secondMDP,
thissolution
produces
error
in
both
B
and
ability of a string of k Bs is 2 k . Now consider
the
value
p
which
generate
the
same
data,
have
di↵erent
BEs.
Thus,
p
p
data
alone;
knowledge
of
the
MDP
beyond
what
d(B)
+
d(B
),
or
2/3
if
the
three
statesareequally
weighted
by
d.
T
of
+ d(B
), or knowledge
2/3 if thestatesare equallyP, this is an exact solution, and the overall BE is ofzero.d(B)In dataalone;of threethe MDPbeyondwhat isweighte
revealMoreover,
the
two
MDPs
have
di↵erent
minim
the
same
data,
have
di↵erent
Thus,
the Thus,
BEcannotbevac
whichBEgenerate
the same
data,
haveBEs.
di↵erentBEs.
the BEfor angenerate
overalloduces an error in both B and B of 1, which
Moreover,thetwo MDPshave
di↵erent
minimal-BEtheminimal-BE
valuefunction
is
the
exact
solutiknowledge
of
the
MDP
beyondwhatisrevealed
in
the
data
is
alone;
knowledge
of
the
MDP
beyondwhatisrevealed
inthree states are equally weighted by d.data
Thealone;
twodata
MDPs,=r
the minimal-BE value function is the exact solution v2 theMoreover,
the
two
MDPs
have
di↵erent
minimal-BE
valuefunctions.
For
Moreover,
the
two
MDPs
have
di↵erent
minimal-BE
value
funct
ve di↵erent BEs. Thus, the BE cannot be estimated from
2. This is a critical observation, as it is possible for an
2. Thisis a criticalas it vis possible
an errorfunct=in 0learning
forforForthminimal-BE
valuefunctionthe observation,
exact
perfectly
satisfactory
for usesettings
becau= 0. forany
the minimal-BE
valueisfunction
is solution
the exact
solutionvany
P beyond what is revealed in the data istherequired.
perfectly satisfactory for use in learning settings because the val
from
data.
For
example,
this
is
what
happenswithth2
from data.
example,thiswhatforhappens
with
VE.toThedi↵erent minimal-BE value functions. For
the isfirst
MDP,
2. Thisa2.criticalobservation,
as itForis possible
anis error
functiontobethe
unobservab
This isa criticalobservation,
as it forispossiblean errorfunctionbefor use in learning
becausethe because
value that
perfectly satisfactory
for use settingsin learning
settings
theminimizes
value thatitmc
secondsatisfactory
MDP,
the exact solution v = 0 for any . For theperfectlyfrom data. For
thisis what thishappenswith
the VE.with
ThetheVEVE.
is notfromexample,
data. For
example,is whathappens
Theobservable
VE is20not278

Chapter 11: O↵-policy Methods with Approximation

11.7

Gradient-TD Methods

We now consider SGD methods for minimizing the PBE. As true SGD methods, these
Gradient-TD methods have robust convergence properties even under o↵-policy training
and nonlinear function approximation. Remember that in the linear case there is always
an exact solution, the TD fixed point wTD , at which the PBE is zero. This solution could
be found by least-squares methods (Section 9.8), but only by methods of quadratic O(d2 )complexity in the number of parameters. We seek instead an SGD method, which shouldbe O(d) and have robust convergence properties. Gradient-TD methods come close toachieving these goals, at the cost of a rough doubling of computational complexity.
To derive an SGD method for the PBE (assuming linear function approximation) we
begin by expanding and rewriting the objective (11.22) in matrix terms:
2

PBE(w) = ⇧ ¯w µ= (⇧ ¯w )> D⇧ ¯w= ¯> ⇧> D⇧ ¯w(from (11.14))w= ¯> DX X> DXw

1

X> D ¯w(11.25)(using (11.13) and the identity ⇧> D⇧ = DX X> DX>= X> D ¯w
X> DX

1

X> D ¯w .

1

X> D)(11.26)

The gradient with respect to w is
⇥
⇤>
X> DX
rPBE(w) = 2r X> D ¯w

1

X> D ¯w .

To turn this into an SGD method, we have to sample something on every time step thathas this quantity as its expected value. Let us take µ to be the distribution of states
visited under the behavior policy. All three of the factors above can then be written interms of expectations under this distribution. For example, the last factor can be written
X
X> D ¯w =µ(s)x(s) ¯w (s) = E[⇢t t xt ] ,swhich is just the expectation of the semi-gradient TD(0) update (11.2). The first factor
is the transpose of the gradient of this update:⇥⇤>rE[⇢t t xt ] = E ⇢t r t> x>t⇥⇤= E ⇢t r(Rt+1 + w> xt+1 w> xt )> x>(using episodic t )t⇥⇤>= E ⇢t ( xt+1 xt )xt .
Finally, the middle factor is the inverse of the expected outer-product matrix of thefeature vectors:
X⇥⇤
X> DX =µ(s)x(s)x(s)> = E xt x>t .
s

11.7. Gradient-TD Methods

279

Substituting these expectations for the three factors in our expression for the gradient of
the PBE, we get
⇥
⇤ ⇥
⇤ 1
>
rPBE(w) = 2E ⇢t ( xt+1 xt )x>
E[⇢t t xt ] .
(11.27)t E xt xt

It might not be obvious that we have made any progress by writing the gradient in thisform. It is a product of three expressions and the first and last are not independent.
They both depend on the next feature vector xt+1 ; we cannot simply sample both ofthese expectations and then multiply the samples. This would give us a biased estimateof the gradient just as in the residual-gradient algorithm.
Another idea would be to estimate the three expectations separately and then combine
them to produce an unbiased estimate of the gradient. This would work, but would
require a lot of computational resources, particularly to store the first two expectations,which are d ⇥ d matrices, and to compute the inverse of the second. This idea can be
improved. If two of the three expectations are estimated and stored, then the third could
be sampled and used in conjunction with the two stored quantities. For example, you
could store estimates of the second two quantities (using the increment inverse-updating
techniques in Section 9.8) and then sample the first expression. Unfortunately, the overall
algorithm would still be of quadratic complexity (of order O(d2 )).
The idea of storing some estimates separately and then combining them with samples
is a good one and is also used in Gradient-TD methods. Gradient-TD methods estimate
and store the product of the second two factors in (11.27). These factors are a d ⇥ dmatrix and a d-vector, so their product is just a d-vector, like w itself. We denote thissecond learned vector as v:⇥⇤ 1v ⇡ E xt x>E[⇢t t xt ] .
(11.28)tThis form is familiar to students of linear supervised learning. It is the solution to a linear
least-squares problem that tries to approximate ⇢t t from the features. The standard
SGD method for incrementally finding the vector v that minimizes the expected squared2error v> xt ⇢t t is known as the Least Mean Square (LMS) rule (here augmented
with an importance sampling ratio):.
vt+1 = vt + ⇢t t vt> xt xt ,where > 0 is another step-size parameter. We can use this method to e↵ectively achieve(11.28) with O(d) storage and per-step computation.
Given a stored estimate vt approximating (11.28), we can update our main parameter
vector wt using SGD methods based on (11.27). The simplest such rule is
1↵rPBE(wt )(the general SGD rule)
2
⇥
⇤ ⇥
⇤ 1
1
>
= wt
↵2E ⇢t ( xt+1 xt )x>
E[⇢t t xt ](from (11.27))t E xt xt
2
⇥
⇤ ⇥
⇤ 1
>
= wt + ↵E ⇢t (xtxt+1 )x>E[⇢t t xt ](11.29)t E xt xt⇥⇤>⇡ wt + ↵E ⇢t (xtxt+1 )xt vt(based on (11.28))

wt+1 = wt⇡ wt + ↵⇢t (xtxt+1 ) x>
t vt .

(sampling)

280

Chapter 11: O↵-policy Methods with Approximation

This algorithm is called GTD2. Note that if the final inner product (x>t vt ) is done first,
then the entire algorithm is of O(d) complexity.
A slightly better algorithm can be derived by doing a few more analytic steps before
substituting in vt . Continuing from (11.29):
⇥
⇤ ⇥
⇤ 1
>
wt+1 = wt + ↵E ⇢t (xtxt+1 )x>E[⇢t t xt ]t E xt xt
⇥
⇤
⇥
⇤
⇥
⇤ 1
= wt + ↵ E ⇢t xt x>
E ⇢t xt+1 x>
E xt x>E[⇢t t xt ]
t
t
t
⇥
⇤
⇥
⇤ ⇥
⇤ 1
>
>
>
= wt + ↵ E x t x t
E ⇢t xt+1 xt E xt xt
E[⇢t t xt ]
⇣
⌘
⇥
⇤
⇥
⇤
1
>
= wt + ↵ E[⇢t t xt ]E ⇢t xt+1 x>
E
x
x
E[⇢
x
]
t t
t t t
t
⇥
⇤
>
⇡ wt + ↵ E[⇢t t xt ]E ⇢t xt+1 xt vt(based on (11.28))⇡ wt + ↵⇢tt xt

xt+1 x>t vt ,(sampling)which again is O(d) if the final product (x>t vt ) is done first. This algorithm is known as
either TD(0) with gradient correction (TDC) or, alternatively, as GTD(0).
Figure 11.5 shows a sample and the expected behavior of TDC on Baird’s counterexample. As intended, the PBE falls to zero, but note that the individual componentsof the parameter vector do not approach zero. In fact, these values are still far from

TDC

10

Expected TDC

w7

w7
5

p
VE

2

p

0

p
VE

w 1 – w6p

PBE

w 1 – w6

PBE

w8

-2.34
0

Steps

w8
1000

0

Sweeps1000

Figure 11.5: The behavior of the TDC algorithm on Baird’s counterexample. On the left isshown a typical single run, and on the right is shown the expected behavior of this algorithm if
the updates are done synchronously (analogous to (11.9), except for the two TDC parameter
vectors). The step sizes were ↵ = 0.005 and = 0.05.

11.8. Emphatic-TD Methods

281

an optimal solution, v̂(s) = 0, for all s, for which w would have to be proportional to(1, 1, 1, 1, 1, 1, 4, 2)> . After 1000 iterations we are still far from an optimal solution, as
we can see from the VE, which remains almost 2. The system is actually converging toan optimal solution, but progress is extremely slow because the PBE is already so close
to zero.
GTD2 and TDC both involve two learning processes, a primary one for w and asecondary one for v. The logic of the primary learning process relies on the secondary
learning process having finished, at least approximately, whereas the secondary learning
process proceeds without being influenced by the first. We call this sort of asymmetricaldependence a cascade. In cascades we often assume that the secondary learning process
is proceeding faster and thus is always at its asymptotic value, ready and accurate toassist the primary learning process. The convergence proofs for these methods often makethis assumption explicitly. These are called two-time-scale proofs. The fast time scale isthat of the secondary learning process, and the slower time scale is that of the primarylearning process. If ↵ is the step size of the primary learning process, and is the step
size of the secondary learning process, then these convergence proofs will typically requirethat in the limit ! 0 and ↵ ! 0.
Gradient-TD methods are currently the most well understood and widely used stableo↵-policy methods. There are extensions to action values and control (GQ, Maei et al.,2010), to eligibility traces (GTD( ) and GQ( ), Maei, 2011; Maei and Sutton, 2010), andto nonlinear function approximation (Maei et al., 2009). There have also been proposed
hybrid algorithms midway between semi-gradient TD and gradient TD (Hackman, 2012;
White and White, 2016). Hybrid-TD algorithms behave like Gradient-TD algorithms instates where the target and behavior policies are very di↵erent, and behave like semigradient algorithms in states where the target and behavior policies are the same. Finally,
the Gradient-TD idea has been combined with the ideas of proximal methods and control
variates to produce more efficient methods (Mahadevan et al., 2014; Du et al., 2017).

11.8

Emphatic-TD Methods

We turn now to the second major strategy that has been extensively explored for obtaining
a cheap and efficient o↵-policy learning method with function approximation. Recall
that linear semi-gradient TD methods are efficient and stable when trained under the
on-policy distribution, and that we showed in Section 9.4 that this has to do with thepositive definiteness of the matrix A (9.11)4 and the match between the on-policy state
distribution µ⇡ and the state-transition probabilities p(s|s, a) under the target policy. Ino↵-policy learning, we reweight the state transitions using importance sampling so thatthey become appropriate for learning about the target policy, but the state distribution
is still that of the behavior policy. There is a mismatch. A natural idea is to somehow
reweight the states, emphasizing some and de-emphasizing others, so as to return the
distribution of updates to the on-policy distribution. There would then be a match,and stability and convergence would follow from existing results. This is the idea of
4 In the o↵-policy case, the matrix A is generally defined as E⇥⇥s⇠b x(s)E x(St+1 )>⇤⇤St = s, At ⇠ ⇡ .

282

Chapter 11: O↵-policy Methods with Approximation

Emphatic-TD methods, first introduced for on-policy training in Section 9.11.
Actually, the notion of “the on-policy distribution” is not quite right, as there are many
on-policy distributions, and any one of these is sufficient to guarantee stability. Consider
an undiscounted episodic problem. The way episodes terminate is fully determined by thetransition probabilities, but there may be several di↵erent ways the episodes might begin.
However the episodes start, if all state transitions are due to the target policy, then thestate distribution that results is an on-policy distribution. You might start close to theterminal state and visit only a few states with high probability before ending the episode.
Or you might start far away and pass through many states before terminating. Both areon-policy distributions, and training on both with a linear semi-gradient method would
be guaranteed to be stable. However the process starts, an on-policy distribution results
as long as all states encountered are updated up until termination.
If there is discounting, it can be treated as partial or probabilistic termination for these
purposes. If = 0.9, then we can consider that with probability 0.1 the process terminates
on every time step and then immediately restarts in the state that is transitioned to. A
discounted problem is one that is continually terminating and restarting with probability
1
on every step. This way of thinking about discounting is an example of a moregeneral notion of pseudo termination—termination that does not a↵ect the sequence ofstate transitions, but does a↵ect the learning process and the quantities being learned.
This kind of pseudo termination is important to o↵-policy learning because the restartingis optional—remember we can start any way we want to—and the termination relieves
the need to keep including encountered states within the on-policy distribution. That is,
if we don’t consider the new states as restarts, then discounting quickly gives us a limited
on-policy distribution.
The one-step Emphatic-TD algorithm for learning episodic state values is defined by:
t = Rt+1 +v̂(St+1 ,wt )v̂(St ,wt ),wt+1 = wt + ↵Mt ⇢t t rv̂(St ,wt ),Mt = ⇢t 1 Mt 1 + It ,with It , the interest, being arbitrary and Mt , the emphasis, being initialized to M 1 = 0.
How does this algorithm perform on Baird’s counterexample? Figure 11.6 shows the
trajectory in expectation of the components of the parameter vector (for the case inwhich It = 1, for all t). There are some oscillations but eventually everything converges
and the VE goes to zero. These trajectories are obtained by iteratively computing the
expectation of the parameter vector trajectory without any of the variance due to samplingof transitions and rewards. We do not show the results of applying the Emphatic-TD
algorithm directly because its variance on Baird’s counterexample is so high that it isnigh impossible to get consistent results in computational experiments. The algorithm
converges to the optimal solution in theory on this problem, but in practice it doesnot. We turn to the topic of reducing the variance of all these algorithms in the next
section.

11.9. Reducing Variance

283

10

w7
p
VE

5
2

w 1 – w6

0

w8
-5
0

Sweeps1000

Figure 11.6: The behavior of the one-step Emphatic-TD algorithm in expectation on Baird’s
counterexample. The step size was ↵ = 0.03.

11.9

Reducing VarianceO↵-policy learning is inherently of greater variance than on-policy learning. This is not
surprising; if you receive data less closely related to a policy, you should expect to learn
less about the policy’s values. In the extreme, one may be able to learn nothing. You
can’t expect to learn how to drive by cooking dinner, for example. Only if the target andbehavior policies are related, if they visit similar states and take similar actions, should
one be able to make significant progress in o↵-policy training.
On the other hand, any policy has many neighbors, many similar policies with considerable overlap in states visited and actions chosen, and yet which are not identical.
The raison d’être of o↵-policy learning is to enable generalization to this vast numberof related-but-not-identical policies. The problem remains of how to make the best useof the experience. Now that we have some methods that are stable in expected value(if the step sizes are set right), attention naturally turns to reducing the variance of theestimates. There are many possible ideas, and we can just touch on a few of them in thisintroductory text.
Why is controlling variance especially critical in o↵-policy methods based on importancesampling? As we have seen, importance sampling often involves products of policy ratios.
The ratios are always one in expectation (5.13), but their actual values may be very highor as low as zero. Successive ratios are uncorrelated, so their products are also always one
in expected value, but they can be of very high variance. Recall that these ratios multiply
the step size in SGD methods, so high variance means taking steps that vary greatly intheir sizes. This is problematic for SGD because of the occasional very large steps. They
must not be so large as to take the parameter to a part of the space with a very di↵erent
gradient. SGD methods rely on averaging over multiple steps to get a good sense of
the gradient, and if they make large moves from single samples they become unreliable.
If the step-size parameter is set small enough to prevent this, then the expected step

284

Chapter 11: O↵-policy Methods with Approximation

can end up being very small, resulting in very slow learning. The notions of momentum(Derthick, 1984), of Polyak-Ruppert averaging (Polyak, 1990; Ruppert, 1988; Polyak andJuditsky, 1992), or further extensions of these ideas may significantly help. Methods foradaptively setting separate step sizes for di↵erent components of the parameter vector
are also pertinent (e.g., Jacobs, 1988; Sutton, 1992b, c), as are the “importance weightaware” updates of Karampatziakis and Langford (2010).
In Chapter 5 we saw how weighted importance sampling is significantly better behaved,with lower variance updates, than ordinary importance sampling. However, adapting
weighted importance sampling to function approximation is challenging and can probably
only be done approximately with O(d) complexity (Mahmood and Sutton, 2015).
The Tree Backup algorithm (Section 7.5) shows that it is possible to perform someo↵-policy learning without using importance sampling. This idea has been extended tothe o↵-policy case to produce stable and more efficient methods by Munos, Stepleton,
Harutyunyan, and Bellemare (2016) and by Mahmood, Yu and Sutton (2017).
Another, complementary strategy is to allow the target policy to be determined inpart by the behavior policy, in such a way that it never can be so di↵erent from it tocreate large importance sampling ratios. For example, the target policy can be defined byreference to the behavior policy, as in the “recognizers” proposed by Precup et al. (2006).

11.10

SummaryO↵-policy learning is a tempting challenge, testing our ingenuity in designing stable andefficient learning algorithms. Tabular Q-learning makes o↵-policy learning seem easy,
and it has natural generalizations to Expected Sarsa and to the Tree Backup algorithm.
But as we have seen in this chapter, the extension of these ideas to significant function
approximation, even linear function approximation, involves new challenges and forces us
to deepen our understanding of reinforcement learning algorithms.
Why go to such lengths? One reason to seek o↵-policy algorithms is to give flexibility
in dealing with the tradeo↵ between exploration and exploitation. Another is to freebehavior from learning, and avoid the tyranny of the target policy. TD learning appears
to hold out the possibility of learning about multiple things in parallel, of using one
stream of experience to solve many tasks simultaneously. We can certainly do this in
special cases, just not in every case that we would like to or as efficiently as we would
like to.
In this chapter we divided the challenge of o↵-policy learning into two parts. The
first part, correcting the targets of learning for the behavior policy, is straightforwardly
dealt with using the techniques devised earlier for the tabular case, albeit at the cost ofincreasing the variance of the updates and thereby slowing learning. High variance willprobably always remains a challenge for o↵-policy learning.
The second part of the challenge of o↵-policy learning emerges as the instabilityof semi-gradient TD methods that involve bootstrapping. We seek powerful function
approximation, o↵-policy learning, and the efficiency and flexibility of bootstrapping11.10.

Summary285

TD methods, but it is challenging to combine all three aspects of this deadly triad in
one algorithm without introducing the potential for instability. There have been several
attempts. The most popular has been to seek to perform true stochastic gradient descent(SGD) in the Bellman error (a.k.a. the Bellman residual). However, our analysis concludes
that this is not an appealing goal in many cases, and that anyway it is impossible toachieve with a learning algorithm—the gradient of the BE is not learnable from experiencethat reveals only feature vectors and not underlying states. Another approach, GradientTD methods, performs SGD in the projected Bellman error. The gradient of the PBE
is learnable with O(d) complexity, but at the cost of a second parameter vector with asecond step size. The newest family of methods, Emphatic-TD methods, refine an old idea
for reweighting updates, emphasizing some and de-emphasizing others. In this way they
restore the special properties that make on-policy learning stable with computationallysimple semi-gradient methods.
The whole area of o↵-policy learning is relatively new and unsettled. Which methods
are best or even adequate is not yet clear. Are the complexities of the new methods
introduced at the end of this chapter really necessary? Which of them can be combinede↵ectively with variance reduction methods? The potential for o↵-policy learning remains
tantalizing, the best way to achieve it still a mystery.

Bibliographical and Historical Remarks
11.1

The first semi-gradient method was linear TD( ) (Sutton, 1988). The name“semi-gradient” is more recent (Sutton, 2015a). Semi-gradient o↵-policy TD(0)with general importance-sampling ratio may not have been explicitly stated until
Sutton, Mahmood, and White (2016), but the action-value forms were introduced
by Precup, Sutton, and Singh (2000), who also did eligibility trace forms of thesealgorithms (see Chapter 12). Their continuing, undiscounted forms have notbeen significantly explored. The n-step forms given here are new.

11.2

The earliest w-to-2w example was given by Tsitsiklis and Van Roy (1996), who
also introduced the specific counterexample in the box on page 263. Baird’s
counterexample is due to Baird (1995), though the version we present here isslightly modified. Averaging methods for function approximation were developedby Gordon (1995, 1996b). Other examples of instability with o↵-policy DP
methods and more complex methods of function approximation are given by
Boyan and Moore (1995). Bradtke (1993) gives an example in which Q-learning
using linear function approximation in a linear quadratic regulation problem
converges to a destabilizing policy.

11.3

The deadly triad was first identified by Sutton (1995b) and thoroughly analyzed
by Tsitsiklis and Van Roy (1997). The name “deadly triad” is due to Sutton(2015a).

11.4

This kind of linear analysis was pioneered by Tsitsiklis and Van Roy (1996; 1997),
including the dynamic programming operator. Diagrams like Figure 11.3 were286

Chapter 11: O↵-policy Methods with Approximation

introduced by Lagoudakis and Parr (2003).
What we have called the Bellman operator, and denoted B⇡ , is more commonly
denoted T ⇡ and called a “dynamic programming operator,” while a generalized
form, denoted T ( ) , is called the “TD( ) operator” (Tsitsiklis and Van Roy, 1996,1997).
11.5

The BE was first proposed as an objective function for dynamic programming by
Schweitzer and Seidmann (1985). Baird (1995, 1999) extended it to TD learning
based on stochastic gradient descent. In the literature, BE minimization is often
referred to as Bellman residual minimization.
The earliest A-split example is due to Dayan (1992). The two forms given here
were introduced by Sutton et al. (2009a).

11.6

The contents of this section are new to this text.

11.7

Gradient-TD methods were introduced by Sutton, Szepesvári, and Maei (2009b).
The methods highlighted in this section were introduced by Sutton et al. (2009a)
and Mahmood et al. (2014). A major extension to proximal TD methods
was developed by Mahadevan et al. (2014). The most sensitive empiricalinvestigations to date of Gradient-TD and related methods are given by Geist
and Scherrer (2014), Dann, Neumann, and Peters (2014), White (2015), andGhiassian, Patterson, White, Sutton, and White (2018). Recent developments inthe theory of Gradient-TD methods are presented by Yu (2017).

11.8

Emphatic-TD methods were introduced by Sutton, Mahmood, and White (2016).
Full convergence proofs and other theory were later established by Yu (2015;2016; Yu, Mahmood, and Sutton, 2017), Hallak, Tamar, and Mannor (2015), andHallak, Tamar, Munos, and Mannor (2016).

Chapter 12

Eligibility Traces
Eligibility traces are one of the basic mechanisms of reinforcement learning. For example,
in the popular TD( ) algorithm, the refers to the use of an eligibility trace. Almost any
temporal-di↵erence (TD) method, such as Q-learning or Sarsa, can be combined witheligibility traces to obtain a more general method that may learn more efficiently.
Eligibility traces unify and generalize TD and Monte Carlo methods. When TD
methods are augmented with eligibility traces, they produce a family of methods spanninga spectrum that has Monte Carlo methods at one end ( = 1) and one-step TD methods
at the other ( = 0). In between are intermediate methods that are often better thaneither extreme method. Eligibility traces also provide a way of implementing Monte Carlo
methods online and on continuing problems without episodes.
Of course, we have already seen one way of unifying TD and Monte Carlo methods: then-step TD methods of Chapter 7. What eligibility traces o↵er beyond these is an elegant
algorithmic mechanism with significant computational advantages. The mechanism isa short-term memory vector, the eligibility trace zt 2 Rd , that parallels the long-term
weight vector wt 2 Rd . The rough idea is that when a component of wt participates in
producing an estimated value, then the corresponding component of zt is bumped up andthen begins to fade away. Learning will then occur in that component of wt if a nonzero
TD error occurs before the trace falls back to zero. The trace-decay parameter 2 [0, 1]
determines the rate at which the trace falls.
The primary computational advantage of eligibility traces over n-step methods is that
only a single trace vector is required rather than a store of the last n feature vectors.
Learning also occurs continually and uniformly in time rather than being delayed andthen catching up at the end of the episode. In addition learning can occur and a↵ect
behavior immediately after a state is encountered rather than being delayed n steps.
Eligibility traces illustrate that a learning algorithm can sometimes be implemented ina di↵erent way to obtain computational advantages. Many algorithms are most naturally
formulated and understood as an update of a state’s value based on events that followthat state over multiple future time steps. For example, Monte Carlo methods (Chapter 5)
update a state based on all the future rewards, and n-step TD methods (Chapter 7)

288

Chapter 12: Eligibility Traces

update based on the next n rewards and state n steps in the future. Such formulations,based on looking forward from the updated state, are called forward views. Forward views
are always somewhat complex to implement because the update depends on later things
that are not available at the time. However, as we show in this chapter it is often possible
to achieve nearly the same updates—and sometimes exactly the same updates—with analgorithm that uses the current TD error, looking backward to recently visited statesusing an eligibility trace. These alternate ways of looking at and implementing learningalgorithms are called backward views. Backward views, transformations between forward
views and backward views, and equivalences between them, date back to the introductionof temporal di↵erence learning but have become much more powerful and sophisticatedsince 2014. Here we present the basics of the modern view.
As usual, first we fully develop the ideas for state values and prediction, then extend
them to action values and control. We develop them first for the on-policy case thenextend them to o↵-policy learning. Our treatment pays special attention to the case of
linear function approximation, for which the results with eligibility traces are stronger.
All these results apply also to the tabular and state aggregation cases because these are
special cases of linear function approximation.

12.1

The -return

In Chapter 7 we defined an n-step return as the sum of the first n rewards plus theestimated value of the state reached in n steps, each appropriately discounted (7.1). The
general form of that equation, for any parameterized function approximator, is.
Gt:t+n = Rt+1 + Rt+2 + · · · + n 1 Rt+n + n v̂(St+n ,wt+n 1 ), 0  t  T n, (12.1)
where v̂(s,w) is the approximate value of state s given weight vector w (Chapter 9), andT is the time of episode termination, if any. We noted in Chapter 7 that each n-stepreturn, for n 1, is a valid update target for a tabular learning update, just as it is foran approximate SGD learning update such as (9.7).
Now we note that a valid update can be done not just toward any n-step return, but
toward any average of n-step returns for di↵erent ns. For example, an update can bedone toward a target that is half of a two-step return and half of a four-step return:1
1
2 Gt:t+2 + 2 Gt:t+4 . Any set of n-step returns can be averaged in this way, even an infinite
set, as long as the weights on the component returns are positive and sum to 1. The
composite return possesses an error reduction property similar to that of individual n-stepreturns (7.3) and thus can be used to construct updates with guaranteed convergenceproperties. Averaging produces a substantial new range of algorithms. For example, one
could average one-step and infinite-step returns to obtain another way of interrelating TD
and Monte Carlo methods. In principle, one could even average experience-based updates
with DP updates to get a simple combination of experience-based and model-based
methods (cf. Chapter 8).
An update that averages simpler component updates is called a compound update. The
backup diagram for a compound update consists of the backup diagrams for each of thecomponent updates with a horizontal line above them and the weighting fractions below.

12.1. The -return289

For example, the compound update for the case mentioned at the start ofthis section, mixing half of a two-step return and half of a four-step return,has the diagram shown to the right. A compound update can only be done
when the longest of its component updates is complete. The update at theright, for example, could only be done at time t + 4 for the estimate formed attime t. In general one would like to limit the length of the longest component
update because of the corresponding delay in the updates.
The TD( ) algorithm can be understood as one particular way of averaging
n-step updates. This average contains all the n-step updates, each weighted
proportionally to n 1 (where 2 [0, 1)), and is normalized by a factor of1
to ensure that the weights sum to 1 (Figure 12.1). The resulting update
is toward a return, called the -return, defined in its state-based form by.
Gt = (1

)

1
X

n 1

Gt:t+n .

(12.2)

n=1

1
2

1
2

Figure 12.2 further illustrates the weighting on the sequence of n-step returns in the-return. The one-step return is given the largest weight, 1; the two-step return isgiven the next largest weight, (1) ; the three-step return is given the weight (1) 2;
and so on. The weight fades by with each additional step. After a terminal state has
been reached, all subsequent n-step returns are equal to the conventional return, Gt . If

TD( )
St
At
St+1 Rt+1
1

At+1

···
(1

St+2 Rt+2

)

At+2

···

(1

X

=1

) 2

··

AT

·

1

ST R TT

t 1

Figure 12.1: The backup diagram for TD( ). If = 0, then the overall update reduces to its
first component, the one-step TD update, whereas if = 1, then the overall update reduces to
its last component, the Monte Carlo update.

290

Chapter 12: Eligibility Traces

weight given to
the 3-step return
) 2
is (1

total area = 1
decay by "WeightWeightingweight given toactual, final return
is T t 1

1!"

T

t
Time

Figure 12.2: Weighting given in the -return to each of the n-step returns.

we want, we can separate these post-termination terms from the main sum, yielding
Gt

= (1

)

TX
t 1

n 1

Gt:t+n +

T

t 1

Gt ,

(12.3)

n=1

as indicated in the figures. This equation makes it clearer what happens when = 1. In
this case the main sum goes to zero, and the remaining term reduces to the conventional
return. Thus, for = 1, updating according to the -return is a Monte Carlo algorithm.
On the other hand, if = 0, then the -return reduces to Gt:t+1 , the one-step return.
Thus, for = 0, updating according to the -return is a one-step TD method.
Exercise 12.1 Just as the return can be written recursively in terms of the first reward anditself one-step later (3.9), so can the -return. Derive the analogous recursive relationshipfrom (12.2) and (12.1).
⇤
Exercise 12.2 The parameter characterizes how fast the exponential weighting in
Figure 12.2 falls o↵, and thus how far into the future the -return algorithm looks indetermining its update. But a rate factor such as is sometimes an awkward way ofcharacterizing the speed of the decay. For some purposes it is better to specify a time
constant, or half-life. What is the equation relating and the half-life, ⌧ , the time bywhich the weighting sequence will have fallen to half of its initial value?
⇤
We are now ready to define our first learning algorithm based on the -return: theo↵-line -return algorithm. As an o↵-line algorithm, it makes no changes to the weightvector during the episode. Then, at the end of the episode, a whole sequence of o↵-line
updates are made according to our usual semi-gradient rule, using the -return as thetarget:hi.
wt+1 = wt + ↵ Gt v̂(St ,wt ) rv̂(St ,wt ), t = 0, . . . , T 1.
(12.4)

12.1. The -return291

The -return gives us an alternative way of moving smoothly between Monte Carloand one-step TD methods that can be compared with the n-step bootstrapping way
developed in Chapter 7. There we assessed e↵ectiveness on a 19-state random walk task
(Example 7.1, page 144). Figure 12.3 shows the performance of the o↵-line -returnalgorithm on this task alongside that of the n-step methods (repeated from Figure 7.2).
The experiment was just as described earlier except that for the -return algorithm wevaried instead of n. The performance measure used is the estimated root-mean-squareerror between the correct and estimated values of each state measured at the end of
the episode, averaged over the first 10 episodes and the 19 states. Note that overall
performance of the o↵-line -return algorithms is comparable to that of the n-stepalgorithms. In both cases we get best performance with an intermediate value of thebootstrapping parameter, n for n-step methods and for the o↵-line -return algorithm.

n-step TD methods

Offline
λ-return algorithm
0.55

0.55

λ=1

λ=.99
λ=.9750.5

RMS error 0.45
at the endof the episode 0.4
over the first
10 episodes 0.35

256
512

(from Chapter 7)

128

n=64

n=32

0.5

λ=.95

Average
0.45
RMS error
over 19 states 0.4
and first 10
episodes λ=0 0.35

λ=.95

0.3

n=32

n=16

0.3

λ=.9

λ=.4

λ=.8

0.25
0

0.2

0.4

0.6

↵

0.8

n=1

n=8
0.25
1

0

0.2

n=2

n=4
0.4

0.6

0.8

↵

Figure 12.3: 19-state Random walk results (Example 7.1): Performance of the o↵-line -returnalgorithm alongside that of the n-step TD methods. In both case, intermediate values of thebootstrapping parameter ( or n) performed best. The results with the o↵-line -return algorithm
are slightly better at the best values of ↵ and , and at high ↵.

The approach that we have been taking so far is what we call the theoretical, or
forward, view of a learning algorithm. For each state visited, we look forward in time toall the future rewards and decide how best to combine them. We might imagine ourselves
riding the stream of states, looking forward from each state to determine its update, assuggested by Figure 12.4. After looking forward from and updating one state, we moveon to the next and never have to work with the preceding state again. Future states,on the other hand, are viewed and processed repeatedly, once from each vantage point
preceding them.

1

292

Chapter 12: Eligibility Traces

RrT

rt+1
R
st+1
t +1 S
+

rt+2
R
st+2
t +2 S+2

Rrtt+3+3 S
s t +3

t+3

t 1

Sstt

T ime

Figure 12.4: The forward view. We decide how to update each state by looking forward to
future rewards and states.

12.2

TD( )

TD( ) is one of the oldest and most widely used algorithms in reinforcement learning.
It was the first algorithm for which a formal relationship was shown between a moretheoretical forward view and a more computationally-congenial backward view using
eligibility traces. Here we will show empirically that it approximates the o↵-line -returnalgorithm presented in the previous section.
TD( ) improves over the o↵-line -return algorithm in three ways. First it updates
the weight vector on every step of an episode rather than only at the end, and thus
its estimates may be better sooner. Second, its computations are equally distributed
in time rather than all at the end of the episode. And third, it can be applied tocontinuing problems rather than just to episodic problems. In this section we present thesemi-gradient version of TD( ) with function approximation.
With function approximation, the eligibility trace is a vector zt 2 Rd with the same
number of components as the weight vector wt . Whereas the weight vector is a long-term
memory, accumulating over the lifetime of the system, the eligibility trace is a short-term
memory, typically lasting less time than the length of an episode. Eligibility traces assist
in the learning process; their only consequence is that they a↵ect the weight vector, andthen the weight vector determines the estimated value.
In TD( ), the eligibility trace vector is initialized to zero at the beginning of theepisode, is incremented on each time step by the value gradient, and then fades away by:.
z 1 = 0,(12.5).
zt =zt 1 + rv̂(St ,wt ), 0  t  T,
where is the discount rate and is the parameter introduced in the previous section,
which we henceforth call the trace-decay parameter. The eligibility trace keeps trackof which components of the weight vector have contributed, positively or negatively, torecent state valuations, where “recent” is defined in terms of . (Recall that in linear
function approximation, rv̂(St ,wt ) is the feature vector, xt , in which case the eligibility
trace vector is just a sum of past, fading, input vectors.) The trace is said to indicate
the eligibility of each component of the weight vector for undergoing learning changes

12.2. TD( )293

should a reinforcing event occur. The reinforcing events we are concerned with are themoment-by-moment one-step TD errors. The TD error for state-value prediction ist.
= Rt+1 + v̂(St+1 ,wt )v̂(St ,wt ).

(12.6)In TD( ), the weight vector is updated on each step proportional to the scalar TD error
and the vector eligibility trace:.
wt+1 = wt + ↵ t zt .

(12.7)

Semi-gradient TD( ) for estimating v̂ ⇡ v⇡
Input: the policy ⇡ to be evaluatedInput: a di↵erentiable function v̂ : S+ ⇥ Rd ! R such that v̂(terminal,·) = 0
Algorithm parameters: step size ↵ > 0, trace decay rate 2 [0, 1]
Initialize value-function weights w arbitrarily (e.g., w = 0)
Loop for each episode:
Initialize S
z0Loop for each step of episode:| Choose A ⇠ ⇡(·|S)| Take action A, observe R, S 0| zz + rv̂(S,w)|
R + v̂(S 0 ,w) v̂(S,w)
| w
w+↵ z
| S
S0
until S 0 is terminalzett
Sst-3-3

zett
s
Stt-2
-2

(a d-dimensional vector)

!t

t

zet
Sstt-1
-1

zett
Sstt

T ime

Sstt+1+1

Figure 12.5: The backward or mechanistic view of TD( ). Each update depends on the current
TD error combined with the current eligibility traces of past events.

294

Chapter 12: Eligibility Traces

TD( ) is oriented backward in time. At each moment we look at the current TD error
and assign it backward to each prior state according to how much that state contributed
to the current eligibility trace at that time. We might imagine ourselves riding along thestream of states, computing TD errors, and shouting them back to the previously visitedstates, as suggested by Figure 12.5. Where the TD error and traces come together, we
get the update given by (12.7), changing the values of those past states for when theyoccur again in the future.
To better understand the backward view of TD( ), consider what happens at various
values of . If
= 0, then by (12.5) the trace at t is exactly the value gradientcorresponding to St . Thus the TD( ) update (12.7) reduces to the one-step semi-gradient
TD update treated in Chapter 9 (and, in the tabular case, to the simple TD rule (6.2)).
This is why that algorithm was called TD(0). In terms of Figure 12.5, TD(0) is thecase in which only the one state preceding the current one is updated by the TD error(other states may have their value estimates changed by generalization due to function
approximation). For larger values of , but still < 1, more of the preceding states are
updated, but each more temporally distant state is updated less because the corresponding
eligibility trace is smaller, as suggested by the figure. We say that the earlier states are
given less credit for the TD error.
If = 1, then the credit given to earlier states falls only by per step. This turns out
to be just the right thing to do to achieve Monte Carlo behavior. For example, rememberthat the TD error, t , includes an undiscounted term of Rt+1 . In passing this back k
steps it needs to be discounted, like any reward in a return, by k , which is just what
the falling eligibility trace achieves. If = 1 and = 1, then the eligibility traces do notdecay at all with time. In this case the method behaves like a Monte Carlo method for
an undiscounted, episodic task. If = 1, the algorithm is also known as TD(1).
TD(1) is a way of implementing Monte Carlo algorithms that is more general than thosepresented earlier and that significantly increases their range of applicability. Whereas
the earlier Monte Carlo methods were limited to episodic tasks, TD(1) can be applied to
discounted continuing tasks as well. Moreover, TD(1) can be performed incrementallyand online. One disadvantage of Monte Carlo methods is that they learn nothing froman episode until it is over. For example, if a Monte Carlo control method takes an actionthat produces a very poor reward but does not end the episode, then the agent’s tendency
to repeat the action will be undiminished during the episode. Online TD(1), on the other
hand, learns in an n-step TD way from the incomplete ongoing episode, where the nsteps are all the way up to the current step. If something unusually good or bad happens
during an episode, control methods based on TD(1) can learn immediately and alter their
behavior on that same episode.
It is revealing to revisit the 19-state random walk example (Example 7.1) to see how
well TD( ) does in approximating the o↵-line -return algorithm. The results for bothalgorithms are shown in Figure 12.6. For each value, if ↵ is selected optimally for it (orsmaller), then the two algorithms perform virtually identically. If ↵ is chosen larger thanis optimal, however, then the -return algorithm is only a little worse whereas TD( )is much worse and may even be unstable. This is not catastrophic for TD( ) on thisproblem, as these higher parameter values are not what one would want to use anyway,
but for other problems it can be a significant weakness.

12.3. n-step Truncated -return Methods

295

Offline
λ-return algorithm

TD(λ)
0.55

1

.99 .975(from the previous section)

λ=1

λ=.95

λ=.9

0.5

λ=.99
λ=.975

λ=.8

λ=.95

RMS error 0.45
at the endof the episode 0.4
over the first
10 episodes 0.35

λ=0

λ=0

λ=.95

0.3

λ=.4

λ=.9

λ=.9

λ=.8

0.25
0

0.2

λ=.4

λ=.8
0.4

↵

0.6

0.8

1

0

0.2

0.4

0.6

0.8

1

↵

Figure 12.6: 19-state Random walk results (Example 7.1): Performance of TD( ) alongsidethat of the o↵-line -return algorithm. The two algorithms performed virtually identically at
low (less than optimal) ↵ values, but TD( ) was worse at high ↵ values.

Linear TD( ) has been proved to converge in the on-policy case if the step-size
parameter is reduced over time according to the usual conditions (2.7). Just as discussed
in Section 9.4, convergence is not to the minimum-error weight vector, but to a nearby
weight vector that depends on . The bound on solution quality presented in that section(9.14) can now be generalized to apply for any . For the continuing discounted case,
VE(w1 ) 11

min VE(w).
w(12.8)

That is, the asymptotic error is no more than 11
times the smallest possible error. As
approaches 1, the bound approaches the minimum error (and it is loosest at = 0).
In practice, however, = 1 is often the poorest choice, as will be illustrated later inFigure 12.14.
Exercise 12.3 Some insight into how TD( ) can closely approximate the o↵-line -returnalgorithm can be gained by seeing that the latter’s error term (in brackets in (12.4)) canbe written as the sum of TD errors (12.6) for a single fixed w. Show this, following thepattern of (6.6), and using the recursive relationship for the -return you obtained in
Exercise 12.1.
⇤
Exercise 12.4 Use your result from the preceding exercise to show that, if the weight
updates over an episode were computed on each step but not actually used to change theweights (w remained fixed), then the sum of TD( )’s weight updates would be the sameas the sum of the o↵-line -return algorithm’s updates.
⇤

12.3

n-step Truncated -return Methods

The o↵-line -return algorithm is an important ideal, but it is of limited utility because
it uses the -return (12.2), which is not known until the end of the episode. In the296

Chapter 12: Eligibility Tracescontinuing case, the -return is technically never known, as it depends on n-step returns
for arbitrarily large n, and thus on rewards arbitrarily far in the future. However, thedependence becomes weaker for longer-delayed rewards, falling by
for each step ofdelay. A natural approximation, then, would be to truncate the sequence after somenumber of steps. Our existing notion of n-step returns provides a natural way to do thisin which the missing rewards are replaced with estimated values.
In general, we define the truncated -return for time t, given data only up to somelater horizon, h, as.
Gt:h = (1

)

hX
t 1

n 1

Gt:t+n +h t 1

0  t < h  T.

Gt:h ,

n=1(12.9)

If you compare this equation with the -return (12.3), it is clear that the horizon h isplaying the same role as was previously played by T , the time of termination. Whereas
in the -return there is a residual weight given to the conventional return Gt , here it isgiven to the longest available n-step return, Gt:h (Figure 12.2).
The truncated -return immediately gives rise to a family of n-step -return algorithms
similar to the n-step methods of Chapter 7. In all of these algorithms, updates aredelayed by n steps and only take into account the first n rewards, but now all the k-stepreturns are included for 1  k  n (whereas the earlier n-step algorithms used only then-step return), weighted geometrically as in Figure 12.2. In the state-value case, thisfamily of algorithms is known as Truncated TD( ), or TTD( ). The compound backup
diagram, shown in Figure 12.7, is similar to that for TD( ) (Figure 12.1) except that thelongest component update is at most n steps rather than always going all the way to then-step truncated TD( )St
At
St+1 Rt+1
1

or,
if t + n

(1

At+1

T

···

···

AT

)

ST R T···(1

At+n 1

) 2

1

T

t 1

St+n Rt+nn 1

Figure 12.7: The backup diagram for Truncated TD( ).

12.4. Redoing Updates: Online -return Algorithm297

end of the episode. TTD( ) is defined by (cf. (9.15)):⇥⇤.
wt+n = wt+n 1 + ↵ Gt:t+n v̂(St ,wt+n 1 ) rv̂(St ,wt+n 1 ),0  t < T.

This algorithm can be implemented efficiently so that per-step computation does not scalewith n (though of course memory must). Much as in n-step TD methods, no updates are
made on the first n 1 time steps of each episode, and n 1 additional updates are madeupon termination. Efficient implementation relies on the fact that the k-step -returncan be written exactly as
Gt:t+k = v̂(St ,wt 1 ) +

t+k
X1

(

)i t i0 ,(12.10)

i=t

where
0 .
t = Rt+1 +v̂(St+1 ,wt )v̂(St ,wt 1 ).

Exercise 12.5 Several times in this book (often in exercises) we have established that
returns can be written as sums of TD errors if the value function is held constant. Whyis (12.10) another instance of this? Prove (12.10).
⇤

12.4

Redoing Updates: Online -return Algorithm

Choosing the truncation parameter n in Truncated TD( ) involves a tradeo↵. n should
be large so that the method closely approximates the o↵-line -return algorithm, but itshould also be small so that the updates can be made sooner and can influence behavior
sooner. Can we get the best of both? Well, yes, in principle we can, albeit at the cost ofcomputational complexity.
The idea is that, on each time step as you gather a new increment of data, you go backand redo all the updates since the beginning of the current episode. The new updates
will be better than the ones you previously made because now they can take into accountthe time step’s new data. That is, the updates are always towards an n-step truncated-return target, but they always use the latest horizon. In each pass over that episode
you can use a slightly longer horizon and obtain slightly better results. Recall that thetruncated -return is defined in (12.9) as.
Gt:h = (1

)

hX
t 1

n 1

Gt:t+n +h t 1

Gt:h .

n=1

Let us step through how this target could ideally be used if computational complexity was
not an issue. The episode begins with an estimate at time 0 using the weights w0 fromthe end of the previous episode. Learning begins when the data horizon is extended totime step 1. The target for the estimate at step 0, given the data up to horizon 1, could
only be the one-step return G0:1 , which includes R1 and bootstraps from the estimatev̂(S1 ,w0 ). Note that this is exactly what G0:1 is, with the sum in the first term of the298

Chapter 12: Eligibility Traces

equation degenerating to zero. Using this update target, we construct w1 . Then, afteradvancing the data horizon to step 2, what do we do? We have new data in the form of
R2 and S2 , as well as the new w1 , so now we can construct a better update target G0:2
for the first update from S0 as well as a better update target G1:2 for the second updatefrom S1 . Using these improved targets, we redo the updates at S1 and S2 , starting again
from w0 , to produce w2 . Now we advance the horizon to step 3 and repeat, going all theway back to produce three new targets, redoing all updates starting from the original w0
to produce w3 , and so on. Each time the horizon is advanced, all the updates are redone
starting from w0 using the weight vector from the preceding horizon.
This conceptual algorithm involves multiple passes over the episode, one at eachhorizon, each generating a di↵erent sequence of weight vectors. To describe it clearly wehave to distinguish between the weight vectors computed at the di↵erent horizons. Let us
use wth to denote the weights used to generate the value at time t in the sequence up tohorizon h. The first weight vector w0h in each sequence is that inherited from the previous
episode (so they are the same for all h), and the last weight vector whh in each sequence
defines the ultimate weight-vector sequence of the algorithm. At the final horizon h = T
we obtain the final weights wTT which will be passed on to form the initial weights of thenext episode. With these conventions, the three first sequences described in the previous
paragraph can be given explicitly:⇥⇤.
h = 1 : w11 = w01 + ↵ G0:1 v̂(S0 ,w01 ) rv̂(S0 ,w01 ),
h=2:

h=3:

⇥
.
w12 = w02 + ↵ G0:2⇥.
w22 = w12 + ↵ G1:2⇥.
w13 = w03 + ↵ G0:3⇥.
w23 = w13 + ↵ G1:3⇥.
w33 = w23 + ↵ G2:3⇤v̂(S0 ,w02 ) rv̂(S0 ,w02 ),⇤v̂(S1 ,w12 ) rv̂(S1 ,w12 ),⇤v̂(S0 ,w03 ) rv̂(S0 ,w03 ),⇤v̂(S1 ,w13 ) rv̂(S1 ,w13 ),⇤v̂(S2 ,w23 ) rv̂(S2 ,w23 ).

The general form for the update is
⇥
⇤
.
h
wt+1
= wth + ↵ Gt:h v̂(St ,wth ) rv̂(St ,wth ), 0  t < h  T.

.
This update, together with wt = wtt defines the online -return algorithm.
The online -return algorithm is fully online, determining a new weight vector wt
at each step t during an episode, using only information available at time t. Its maindrawback is that it is computationally complex, passing over the portion of the episodeexperienced so far on every step. Note that it is strictly more complex than the o↵-line-return algorithm, which passes through all the steps at the time of termination but does
not make any updates during the episode. In return, the online algorithm can be expected
to perform better than the o↵-line one, not only during the episode when it makes an
update while the o↵-line algorithm makes none, but also at the end of the episode because
the weight vector used in bootstrapping (in Gt:h ) has had a larger number of informative

12.5. True Online TD( )299

updates. This e↵ect can be seen if one looks carefully at Figure 12.8, which compares thetwo algorithms on the 19-state random walk task.

Offline
λ-return algorithm

On-line λ-return algorithm= true online TD(λ)

0.55(from Section 12.1)

λ=1

λ=1

λ=.99

λ=.99

0.5

λ=.975λ=.975RMS error 0.45
at the endof the episode 0.4
over the first
10 episodes 0.35

λ=.95
λ=.95

λ=0

λ=0

λ=.95

λ=.95
0.3

λ=.9
0

0.2

0.4

λ=.9

λ=.4

λ=.8

0.25

↵

0.6

0.8

λ=.4

λ=.8
1

0

0.2

0.4

0.6

0.8

1

↵

Figure 12.8: 19-state Random walk results (Example 7.1): Performance of online and o↵-line-return algorithms. The performance measure here is the VE at the end of the episode, whichshould be the best case for the o↵-line algorithm. Nevertheless, the online algorithm performs
subtly better. For comparison, the = 0 line is the same for both methods.

12.5

True Online TD( )The online -return algorithm just presented is currently the best performing temporaldi↵erence algorithm. It is an ideal which online TD( ) only approximates. As presented,
however, the online -return algorithm is very complex. Is there a way to invert thisforward-view algorithm to produce an efficient backward-view algorithm using eligibility
traces? It turns out that there is indeed an exact computationally congenial implementation of the online -return algorithm for the case of linear function approximation. This
implementation is known as the true online TD( ) algorithm because it is “truer” to theideal of the online -return algorithm than the TD( ) algorithm is.
The derivation of true online TD( ) is a little too complex to present here (see thenext section and the appendix to the paper by van Seijen et al., 2016) but its strategy issimple. The sequence of weight vectors produced by the online -return algorithm canbe arranged in a triangle:
w00
w01
w02
w03
..
.

w11
w12
w13
..
.

w22
w23
..
.

w33
..
.

w0T

w1T

w2T

w3T

..

.
···

wTT

One row of this triangle is produced on each time step. It turns out that the weight vectors
on the diagonal, the wtt , are the only ones really needed. The first, w00 , is the initial weight

300

Chapter 12: Eligibility Traces

vector of the episode, the last, wTT , is the final weight vector, and each weight vector
along the way, wtt , plays a role in bootstrapping in the n-step returns of the updates.
In the final algorithm the diagonal weight vectors are renamed without a superscript,.
wt = wtt . The strategy then is to find a compact, efficient way of computing each wtt
from the one before. If this is done, for the linear case in which v̂(s,w) = w> x(s), thenwe arrive at the true online TD( ) algorithm:.
wt+1 = wt + ↵ t zt + ↵ wt> xt wt> 1 xt (zt xt ),.
where we have used the shorthand xt = x(St ), t is defined as in TD( ) (12.6), and zt isdefined by.
zt =zt 1 + 1 ↵ z>
(12.11)t 1 xt xt .
This algorithm has been proven to produce exactly the same sequence of weight vectors,
wt , 0  t  T , as the online -return algorithm (van Seijen et al. 2016). Thus the resultson the random walk task on the left of Figure 12.8 are also its results on that task. Now,
however, the algorithm is much less expensive. The memory requirements of true online
TD( ) are identical to those of conventional TD( ), while the per-step computation isincreased by about 50% (there is one more inner product in the eligibility-trace update).
Overall, the per-step computational complexity remains of O(d), the same as TD( ).
Pseudocode for the complete algorithm is given in the box.
True online TD( ) for estimating w> x ⇡ v⇡
Input: the policy ⇡ to be evaluatedInput: a feature function x : S+ ! Rd such that x(terminal, ·) = 0
Algorithm parameters: step size ↵ > 0, trace decay rate 2 [0, 1]
Initialize value-function weights w 2 Rd (e.g., w = 0)
Loop for each episode:
Initialize state and obtain initial feature vector xz0(a d-dimensional vector)
Vold0(a temporary scalar variable)
Loop for each step of episode:| Choose A ⇠ ⇡| Take action A, observe R, x0 (feature vector of the next state)
| V
w> x
0
| V
w > x0|
R+ V0 V| zz + 1 ↵ z> x x| w
w + ↵( + V Vold )z ↵(V Vold )x| VoldV00| xx
until x0 = 0 (signaling arrival at a terminal state)
The eligibility trace (12.11) used in true online TD( ) is called a dutch trace todistinguish it from the trace (12.5) used in TD( ), which is called an accumulating trace.

12.6. *Dutch Traces in Monte Carlo Learning

301

Earlier work often used a third kind of trace called the replacing trace, defined only forthe tabular case or for binary feature vectors such as those produced by tile coding. The
replacing trace is defined on a component-by-component basis depending on whether thecomponent of the feature vector was 1 or 0:.
zi,t =⇢

1zi,t 1

if xi,t = 1
otherwise.

(12.12)

Nowadays, we see replacing traces as crude approximations to dutch traces, which largely
supersede them. Dutch traces usually perform better than replacing traces and have aclearer theoretical basis. Accumulating traces remain of interest for nonlinear function
approximations where dutch traces are not available.

12.6*Dutch Traces in Monte Carlo Learning

Although eligibility traces are closely associated historically with TD learning, in fact
they have nothing to do with it. In fact, eligibility traces arise even in Monte Carlolearning, as we show in this section. We show that the linear MC algorithm (Chapter 9),
taken as a forward view, can be used to derive an equivalent yet computationally cheaperbackward-view algorithm using dutch traces. This is the only equivalence of forward- andbackward-views that we explicitly demonstrate in this book. It gives some of the flavorof the proof of equivalence of true online TD( ) and the online -return algorithm, but ismuch simpler.
The linear version of the gradient Monte Carlo prediction algorithm (page 202) makes
the following sequence of updates, one for each time step of the episode:⇥.
wt+1 = wt + ↵ G⇤
wt> xt xt ,0  t < T.

(12.13)To simplify the example, we assume here that the return G is a single reward received at
the end of the episode (this is why G is not subscripted by time) and that there is no
discounting. In this case the update is also known as the Least Mean Square (LMS) rule.
As a Monte Carlo algorithm, all the updates depend on the final reward/return, so none
can be made until the end of the episode. The MC algorithm is an o↵-line algorithm and
we do not seek to improve this aspect of it. Rather we seek merely an implementation ofthis algorithm with computational advantages. We will still update the weight vector
only at the end of the episode, but we will do some computation during each step of theepisode and less at its end. This will give a more equal distribution of computation—O(d)per step—and also remove the need to store the feature vectors at each step for use laterat the end of each episode. Instead, we will introduce an additional vector memory, theeligibility trace, keeping in it a summary of all the feature vectors seen so far. This will
be sufficient to efficiently recreate exactly the same overall update as the sequence of MC

302

Chapter 12: Eligibility Traces

updates (12.13), by the end of the episode:wT = wT= wT

1+↵

1 + ↵xT= I

wT> 1 xT

G
1>
1 xT 1↵xT

1x>
T 1 wTwTxT

1+ ↵GxT1

1 + ↵GxT

1

1= FT 1 wT 1 + ↵GxT 1.
where Ft = I ↵xt x>
t is a forgetting, or fading, matrix. Now, recursing,
= FT

1 (FT

= FT

1 FT

2 wT

2 + ↵G (FT

= FT

1 FT

2 (FT

3 wT

= FT
..
.

1 FT

2 FT

= FT
|

1 FT

2 · · · F0 w0

= aT

2 wT

{z

aT

2 + ↵GxT

3 wT

1

1 + ↵GzT

1,2 ) + ↵GxT
1 xT

3 + ↵GxT
3 + ↵G (FT

}

+ ↵G

T
X1

1

2 + xT1)3 ) + ↵G (FT
1 FT

FT

2 xT

1 FT

k=0

|

1 xT

3 + FT

2 + xT
1 xT

1)

2 + xT

1)

2 · · · Fk+1 xk

{z
zT 1

}

(12.14)

where aT 1 and zT 1 are the values at time T 1 of two auxiliary memory vectors thatcan be updated incrementally without knowledge of G and with O(d) complexity per time
step. The zt vector is in fact a dutch-style eligibility trace. It is initialized to z0 = x0
and then updated according to
t

. X
zt =
Ft Ft 1 · · · Fk+1 xk ,
k=0

=

t 1
X

k=0

= Ft

1t<T

Ft Ft 1 · · · Fk+1 xk + xt

t 1
X

k=0

Ft 1 Ft 2 · · · Fk+1 xk + xt

= F t zt 1 + x t= I↵xt x>
t zt 1 + x t= zt 1↵xt x>
t zt 1 + x t= zt 1↵ z>
t 1 xt xt + xt

= zt 1 + 1↵z>
t 1 xt xt ,which is the dutch trace for the case of= 1 (cf. Eq. 12.11). The at auxiliary vector is
initialized to a0 = w0 and then updated according to.
at = Ft Ft 1 · · · F0 w0 = Ft at 1 = at 1 ↵xt x>
1  t < T.
t at 1 ,12.7. Sarsa( )303

The auxiliary vectors, at and zt , are updated on each time step t < T and then, at time
T when G is observed, they are used in (12.14) to compute wT . In this way we achieve
exactly the same final result as the MC/LMS algorithm that has poor computational
properties (12.13), but now with an incremental algorithm whose time and memory
complexity per step is O(d). This is surprising and intriguing because the notion ofan eligibility trace (and the dutch trace in particular) has arisen in a setting without
temporal-di↵erence (TD) learning (in contrast to van Seijen and Sutton, 2014). It seems
eligibility traces are not specific to TD learning at all; they are more fundamental thanthat. The need for eligibility traces seems to arise whenever one tries to learn long-term
predictions in an efficient manner.

12.7

Sarsa( )Very few changes in the ideas already presented in this chapter are required in order toextend eligibility-traces to action-value methods. To learn approximate action values,q̂(s, a, w), rather than approximate state values, v̂(s,w), we need to use the action-value
form of the n-step return, from Chapter 10:.
Gt:t+n = Rt+1 + · · · +n 1

Rt+n +nq̂(St+n , At+n , wt+n 1 ),t + n < T,.
with Gt:t+n = Gt if t + n
T . Using this, we can form the action-value form of the-return, which is otherwise identical to the state-value form (12.3). The action-value
form of the o↵-line -return algorithm (12.4) simply uses q̂ rather than v̂:hi.
wt+1 = wt + ↵ Gt q̂(St , At , wt ) rq̂(St , At , wt ), t = 0, . . . , T 1,
(12.15).
where Gt = Gt:1 . The compound backup diagram for this forward view is shown in
Figure 12.9. Notice the similarity to the diagram of the TD( ) algorithm (Figure 12.1).
The first update looks ahead one full step, to the next state–action pair, the second looks
ahead two steps, to the second state–action pair, and so on. A final update is based on
the complete return. The weighting of each n-step update in the -return is just as inTD( ) and the -return algorithm (12.3).
The temporal-di↵erence method for action values, known as Sarsa( ), approximatesthis forward view. It has the same update rule as given earlier for TD( ):.
wt+1 = wt + ↵ t zt ,except, naturally, using the action-value form of the TD error:.

t = Rt+1 +q̂(St+1 , At+1 , wt )q̂(St , At , wt ),and the action-value form of the eligibility trace:.
z 1 = 0,.
zt =zt 1 + rq̂(St , At , wt ),0  t  T.

(12.16)

304

Chapter 12: Eligibility Traces

Sarsa( )S t AtSt+1 Rt+1
At+1
1
St+2 Rt+2

···
(1

At+2

)

···

(1

X

=1

) 2

··

AT

·

1

ST R TT

t 1

Figure 12.9: Sarsa( )’s backup diagram. Compare with Figure 12.1.

Complete pseudocode for Sarsa( ) with linear function approximation, binary features,and either accumulating or replacing traces is given in the box on the next page. This
pseudocode highlights a few optimizations possible in the special case of binary features(features are either active (=1) or inactive (=0)).
Example 12.1: Traces in Gridworld The use of eligibility traces can substantially
increase the efficiency of control algorithms over one-step methods and even over n-stepmethods. The reason for this is illustrated by the gridworld example below.
Path taken

G

Action values increased
Path taken
by one-step
Sarsa

G

Action values increased
Sarsa
byone-step
10-step
Sarsa
by bySarsa() with =0.9G
G

Action values increasedby Sarsa( h) with h=0.9

G

The first panel shows the path taken by an agent in a single episode. The initial estimatedvalues were zero, and all rewards were zero except for a positive reward at the goal
location marked by G. The arrows in the other panels show, for various algorithms, which
action-values would be increased, and by how much, upon reaching the goal. A one-stepmethod would increment only the last action value, whereas an n-step method would
equally increment the last n actions’ values (assuming = 1), and an eligibility trace
method would update all the action values up to the beginning of the episode, to di↵erent
degrees, fading with recency. The fading strategy is often the best.

12.7. Sarsa( )305

Sarsa( ) with binary features and linear function approximationfor estimating w> x ⇡ q⇡ or q⇤
Input: a function F(s, a) returning the set of (indices of) active features for s, aInput: a policy ⇡Algorithm parameters: step size ↵ > 0, trace decay rate 2 [0, 1], small " > 0
Initialize: w = (w1 , . . . , wd )> 2 Rd (e.g., w = 0), z = (z1 , . . . , zd )> 2 Rd
Loop for each episode:
Initialize S
Choose A ⇠ ⇡(·|S) or "-greedy according to q̂(S, ·, w)z0Loop for each step of episode:Take action A, observe R, S 0
R
Loop for i in F(S, A):wizizi + 1(accumulating traces)or zi
1(replacing traces)
If S 0 is terminal then:ww+↵ z
Go to next episode
Choose A0 ⇠ ⇡(·|S 0 ) or "-greedy according to q̂(S 0 , ·, w)
Loop for i in F(S 0 , A0 ):
+ wi
w
w+↵ z
z
z
S
S0; A
A0

Exercise 12.6 Modify the pseudocode for Sarsa( ) to use dutch traces (12.11) without theother distinctive features of a true online algorithm. Assume linear function approximation
and binary features.
⇤
Example 12.2: Sarsa( ) on Mountain Car Figure 12.10 (left) on the next pageshows results with Sarsa( ) on the Mountain Car task introduced in Example 10.1. The
function approximation, action selection, and environmental details were exactly as in
Chapter 10, and thus it is appropriate to numerically compare these results with the
Chapter 10 results for n-step Sarsa (right side of the figure). The earlier results varied the
update length n whereas here for Sarsa( ) we vary the trace parameter , which plays
a similar role. The fading-trace bootstrapping strategy of Sarsa( ) appears to result in
more efficient learning on this problem.
There is also an action-value version of our ideal TD method, the online -return algorithm (Section 12.4) and its efficient implementation as true online TD( ) (Section 12.5).
Everything in Section 12.4 goes through without change other than to use the action-value
form of the n-step return given at the beginning of the current section. The analyses in
Sections 12.5 and 12.6 also carry through for action values, the only change being the use

306

Chapter 12: Eligibility Traces
True Online TD( )Sarsa(λ) with replacing
tracestrue online
TD(λ) − Task 1

n-step Sarsa

TD(λ), replacing traces − Task 1
300

1

0.8

280

λ=0

0.4

Mountain Car

Steps per episode

1.5

0
0

n=1

n=16
n=8

0.8

RMS
error 0.6

0.2

300

1

λ=1

averagedover
0.5
step−size
first 50 episodes
and 100 runs

λ=.84

260

280

λ=0

RMS
error 0.6

λ=.99

240

0.2

λ=.92

1

λ=1

0.5

1.5
n=8

1

step−size

220

n=2

240 n=16

λ=0

0
0

1.5

n=4
λ=.98260

0.4

220

λ=.68

0

0.5

n=4

n=1

n=2
1

1.5

1

1.5

λ=.84

200
TD(λ), replacing traces − Task 2

true online TD(λ) − Task 2

λ=.96
1

1

180

0.8

λ=.92

0.8

0

RMS
error 0.6

0.5

1

RMS

0

1.5

(8)
↵ ×errornumber ofλ =tilings
0

0.4

0.5

↵ × number of tilings (8)

0.6

0.4

Figure 12.10: Early performance on the Mountain Car task of Sarsa( ) with replacing traces
0.2
0.2
and n-step Sarsa (copied from Figure 10.4) as a function of the step size, ↵.
1.5

0
0

0.5

1

step−size

1.5

0
0

λ=1
0.5

1

1.5

step−sizeof state–action feature vectors xt = x(St , At ) instead of state feature vectors xt = x(St ).

he end of eachPseudocode
episode, averaged
first 10 episodes,
well as 100 independent
runs,
for Sarsa( ) is given infor overthe theresultingefficientasalgorithm,
called true
onlinethe box on the next page. The figure below compares the performance of various versionsof Sarsa( ) on the Mountain Car example.

averaged overfirst 20 episodes
and 100 runs

return( ) with the traditional−150
dard mountain car task
ngs of each 10⇥10 tiles.
−200↵ = ↵0 /10, for ↵0 from−250
ring/no clearing refers
selected actions are set−300), in case ofMountainreplacing Car
perisepisode −350
true onlineReward
principle

7UXHRQOLQH6DUVD λSarsa(λ) with replacing traces
Sarsa(λ), replacing, clearing

Sarsa(λ) with replacing traces
and clearing the traces of other actions
Sarsa(λ), replacing, no clearing

−400

−450

Sarsa(λ) with accumulating tracesnline version of the for−500
Sarsa(λ), accumulating
ical and intuitive foun−550
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
2
addition, we have preα0with the same compuof tilings (8)↵ × number
l algorithm, which we
Figure4. Average
return over
20 episodes
on mountain
FigureSummary
comparisonof first
Sarsa() algorithms
on thecar
Mountain Car task. Truethat true online
TD(12.11:
)
task
for
=
0.9
and
different↵
0 . Results are averaged over 100
Sarsa( ) performed better than regular Sarsa( ) with both accumulating and replacing
ew exactly, online
in contrast
independent runs.
traces. Also
only approximates
its included is a version of Sarsa( ) with replacing traces in which, on each time step,
the traces for the state and the actions not selected were set to zero.
monstrated empiricallyconventional TD( ) on
ms, by adhering more
Acknowledgements
matching an intuitively
The authors thank Hado van Hasselt and Rupam Mahmood
ne case—that we have

12.8. Variable

and

307

True online Sarsa( ) for estimating w> x ⇡ q⇡ or q⇤
Input: a feature function x : S+ ⇥ A ! Rd such that x(terminal, ·) = 0
Input: a policy ⇡ (if estimating q⇡ )Algorithm parameters: step size ↵ > 0, trace decay rate 2 [0, 1], small " > 0
Initialize: w 2 Rd (e.g., w = 0)
Loop for each episode:
Initialize S
Choose A ⇠ ⇡(·|S) or "-greedy according to q̂(S, ·, w)xx(S, A)z0
Qold
0
Loop for each step of episode:| Take action A, observe R, S 0| Choose A0 ⇠ ⇡(·|S 0 ) or "-greedy according to q̂(S 0 , ·, w)| x0
x(S 0 , A0 )
| Q
w> x
0
| Q
w > x0|R + Q0 Q| z
z + 1 ↵ z> x x| w
w + ↵( + Q Qold )z ↵(Q Qold )x| Qold
Q0
0
| x
x
| A
A0
0
until S is terminal

Finally, there is also a truncated version of Sarsa( ), called forward Sarsa( ) (vanSeijen, 2016), which appears to be a particularly e↵ective model-free control method foruse in conjunction with multi-layer artificial neural networks.

12.8

VariableandWe are starting now to reach the end of our development of fundamental TD learningalgorithms. To present the final algorithms in their most general forms, it is useful togeneralize the degree of bootstrapping and discounting beyond constant parameters tofunctions potentially dependent on the state and action. That is, each time step will havea di↵erent and , denoted t and t . We change notation now so that : S ⇥ A ! [0, 1].
is now a function from states and actions to the unit interval such that t = (St , At ), and.
similarly, : S ! [0, 1] is a function from states to the unit interval such that t = (St ).
Introducing the function , the termination function, is particularly significant because
it changes the return, the fundamental random variable whose expectation we seek to308

Chapter 12: Eligibility Traces

estimate. Now the return is defined more generally as.
Gt = Rt+1 + t+1 Gt+1
= Rt+1 +=1X

t+1 Rt+2 +

k
Y

i=t+1

k=t

i

!

t+1 t+2 Rt+3 +t+1 t+2 t+3 Rt+4 + · · ·Rk+1 ,(12.17)

Q1
where, to assure the sums are finite, we require that k=t k = 0 with probability one forall t. One convenient aspect of this definition is that it enables the episodic setting and
its algorithms to be presented in terms of a single stream of experience, without special
terminal states, start distributions, or termination times. An erstwhile terminal state
becomes a state at which (s) = 0 and which transitions to the start distribution. In thatway (and by choosing (·) as a constant in all other states) we can recover the classicalepisodic setting as a special case. State dependent termination includes other prediction
cases such as pseudo termination, in which we seek to predict a quantity without altering
the flow of the Markov process. Discounted returns can be thought of as such a quantity,
in which case state-dependent termination unifies the episodic and discounted-continuingcases. (The undiscounted-continuing case still needs some special treatment.)
The generalization to variable bootstrapping is not a change in the problem, like
discounting, but a change in the solution strategy. The generalization a↵ects the returns for states and actions. The new state-based -return can be written recursivelyas.
s
Gt s = Rt+1 + t+1 (1(12.18)t+1 )v̂(St+1 ,wt ) + t+1 Gt+1 ,where now we have added the “s” to the superscript to remind us that this is a returnthat bootstraps from state values, distinguishing it from returns that bootstrap fromaction values, which we present below with “a” in the superscript. This equation saysthat the -return is the first reward, undiscounted and una↵ected by bootstrapping, plus
possibly a second term to the extent that we are not discounting at the next state (thatis, according to t+1 ; recall that this is zero if the next state is terminal). To the extent
that we aren’t terminating at the next state, we have a second term which is itself divided
into two cases depending on the degree of bootstrapping in the state. To the extent weare bootstrapping, this term is the estimated value at the state, whereas, to the extent
that we are not bootstrapping, the term is the -return for the next time step. The
action-based -return is either the Sarsa form⇣⌘.
a
Gt a = Rt+1 + t+1 (1(12.19)t+1 )q̂(St+1 , At+1 , wt ) + t+1 Gt+1 ,or the Expected Sarsa form,⇣.
Gt a = Rt+1 + t+1 (1t+1 )V̄t (St+1 ) +⌘a
Gt+1 t+1 ,where (7.8) is generalized to function approximation as. X
V̄t (s) =⇡(a|s)q̂(s, a, wt ).
a(12.20)(12.21)

12.9. O↵-policy Traces with Control Variates

309

Exercise 12.7 Generalize the three recursive equations above to their truncated versions,sadefining Gt:hand Gt:h.
⇤12.9O↵-policy Traces with Control Variates

The final step is to incorporate importance sampling. For methods using non-truncated-returns, there is not a practical option in which the importance-sampling weighting isapplied to the target return (as there is for n-step methods as explained in Section 7.3).
Instead, we move directly to the bootstrapping generalization of per-decision importancesampling with control variates (Section 7.4).
In the state case, our final definition of the -return generalizes (12.18), after the modelof (7.13), to⇣⌘.
s
Gt s = ⇢t Rt+1 + t+1 (1+ (1 ⇢t )v̂(St ,wt ), (12.22)
t+1 )v̂(St+1 ,wt ) + t+1 Gt+1
t |St )
where ⇢t = ⇡(Ab(At |St ) is the usual single-step importance sampling ratio. Much like theother returns we have seen in this book, this final -return can be approximated simply
in terms of sums of the state-based TD error,s .
t = Rt+1 +t+1 v̂(St+1 ,wt )v̂(St ,wt ),(12.23)i i ⇢i ,(12.24)as
Gt s ⇡ v̂(St ,wt ) + ⇢t

1
X
k=t

s
k

k
Y

i=t+1

with the approximation becoming exact if the approximate value function does not change.
Exercise 12.8 Prove that (12.24) becomes exact if the value function does not change.
.
To save writing, consider the case of t = 0, and use the notation Vk = v̂(Sk ,w).
⇤s
Exercise 12.9 The truncated version of the general o↵-policy return is denoted Gt:h.
Guess the correct equation, based on (12.24).
⇤
The above form of the -return (12.24) is convenient to use in a forward-view update,

wt+1 = wt + ↵ Gt s
⇡ wt + ↵⇢tv̂(St ,wt ) rv̂(St ,wt )
!
1
k
X
Y
s
i i ⇢i rv̂(St ,wt ),
k
k=t

i=t+1

which to the experienced eye looks like an eligibility-based TD update—the product islike an eligibility trace and it is multiplied by TD errors. But this is just one time step of
a forward view. The relationship that we are looking for is that the forward-view update,summed over time, is approximately equal to a backward-view update, summed overtime (this relationship is only approximate because again we ignore changes in the value

310

Chapter 12: Eligibility Traces

function). The sum of the forward-view update over time is
1
X

(wt+1

t=0

wt ) ⇡=1 X
1X↵⇢t ks rv̂(St ,wt )t=0 k=t

1 X
k
X

=

1
X

k
Y

i i ⇢ii=t+1(using the summation rule:

↵ ks

k
X
t=0

k=0

i i ⇢ii=t+1↵⇢t rv̂(St ,wt ) ks

k=0 t=0kY⇢t rv̂(St ,wt )

k
Y

Py

i i ⇢i ,

t=x

Py

k=t =

Py

k=x

Pk

t=x )

i=t+1

which would be in the form of the sum of a backward-view TD update if the entire
expression from the second sum on could be written and updated incrementally as aneligibility trace, which we now show can be done. That is, we show that if this expression
was the trace at time k, then we could update it from its value at time k 1 by:
zk =

k
X
t=0

=

k
X1
t=0

=

⇢t rv̂(St ,wt )⇢t rv̂(St ,wt )k k ⇢k

k
X1
t=0

= ⇢k

|

k
Y

i i ⇢ii=t+1
kYi i ⇢ii=t+1⇢t rv̂(St ,wt )
{z

zk

kY1

i=t+1

1

k k zk 1 + rv̂(Sk ,wk ) ,+ ⇢k rv̂(Sk ,wk )i i ⇢i+ ⇢k rv̂(Sk ,wk )}which, changing the index from k to t, is the general accumulating trace update for statevalues:.
z t = ⇢tt t zt 1 + rv̂(St ,wt ) ,(12.25)This eligibility trace, together with the usual semi-gradient parameter-update rule forTD( ) (12.7), forms a general TD( ) algorithm that can be applied to either on-policy oro↵-policy data. In the on-policy case, the algorithm is exactly TD( ) because ⇢t is alway
1 and (12.25) becomes the usual accumulating trace (12.5) (extended to variable and). In the o↵-policy case, the algorithm often works well but, as a semi-gradient method,
is not guaranteed to be stable. In the next few sections we will consider extensions of it
that do guarantee stability.
A very similar series of steps can be followed to derive the o↵-policy eligibility traces
for action-value methods and corresponding general Sarsa( ) algorithms. One could startwith either recursive form for the general action-based -return, (12.19) or (12.20), but
the latter (the Expected Sarsa form) works out to be simpler. We extend (12.20) to the12.9. O↵-policy Traces with Control Variates311

o↵-policy case after the model of (7.14) to produce
⇣
⇤⌘
⇥
.
a
Gt a = Rt+1 + t+1 (1 t+1 )V̄t (St+1 )+ t+1 ⇢t+1 Gt+1
+ V̄t (St+1 ) ⇢t+1 q̂(St+1 , At+1 , wt )⇣⇥ a⇤⌘
= Rt+1 + t+1 V̄t (St+1 ) + t+1 ⇢t+1 Gt+1q̂(St+1 , At+1 , wt )(12.26)

where V̄t (St+1 ) is as given by (12.21). Again the -return can be written approximately
as the sum of TD errors,
Gt a ⇡ q̂(St , At , wt ) +

1
X
k=t

a
k

k
Y

i i ⇢i ,(12.27)i=t+1

using the expectation form of the action-based TD error:at = Rt+1 +t+1 V̄t (St+1 )q̂(St , At , wt ).

(12.28)As before, the approximation becomes exact if the approximate value function does notchange.
Exercise 12.10 Prove that (12.27) becomes exact if the value function does not change.
To save writing, consider the case of t = 0, and use the notation Qk = q̂(Sk , Ak , w). Hint:
Start by writing out 0a and G0 a , then G0 a Q0 .
⇤a
Exercise 12.11 The truncated version of the general o↵-policy return is denoted Gt:h.
Guess the correct equation for it, based on (12.27).
⇤
Using steps entirely analogous to those for the state case, one can write a forward-view
update based on (12.27), transform the sum of the updates using the summation rule,and finally derive the following form for the eligibility trace for action values:.
zt =t t ⇢t zt 1 + rq̂(St , At , wt ).

(12.29)This eligibility trace, together with the expectation-based TD error (12.28) and the usual
semi-gradient parameter-update rule (12.7), forms an elegant, efficient Expected Sarsa( )algorithm that can be applied to either on-policy or o↵-policy data. It is probably thebest algorithm of this type at the current time (though of course it is not guaranteed to
be stable until combined in some way with one of the methods presented in the following
sections). In the on-policy case with constant and , and the usual state–action TD
error (12.16), the algorithm would be identical to the Sarsa( ) algorithm presented in
Section 12.7.
Exercise 12.12 Show in detail the steps outlined above for deriving (12.29) from (12.27).
Start with the update (12.15), substitute Gt a from (12.26) for Gt , then follow similar
steps as led to (12.25).
⇤
At = 1, these algorithms become closely related to corresponding Monte Carloalgorithms. One might expect that an exact equivalence would hold for episodic problems
and o↵-line updating, but in fact the relationship is subtler and slightly weaker than that.
Under these most favorable conditions still there is not an episode by episode equivalence of updates, only of their expectations. This should not be surprising as these methods

312

Chapter 12: Eligibility Traces

make irrevocable updates as a trajectory unfolds, whereas true Monte Carlo methods
would make no update for a trajectory if any action within it has zero probability under
the target policy. In particular, all of these methods, even at = 1, still bootstrap inthe sense that their targets depend on the current value estimates—it’s just that thedependence cancels out in expected value. Whether this is a good or bad property inpractice is another question. Recently, methods have been proposed that do achieve anexact equivalence (Sutton, Mahmood, Precup and van Hasselt, 2014). These methods
require an additional vector of “provisional weights” that keep track of updates whichhave been made but may need to be retracted (or emphasized) depending on the actions
taken later. The state and state–action versions of these methods are called PTD( ) andPQ( ) respectively, where the ‘P’ stands for Provisional.
The practical consequences of all these new o↵-policy methods have not yet been
established. Undoubtedly, issues of high variance will arise as they do in all o↵-policy
methods using importance sampling (Section 11.9).
If < 1, then all these o↵-policy algorithms involve bootstrapping and the deadly
triad applies (Section 11.3), meaning that they can be guaranteed stable only for thetabular case, for state aggregation, and for other limited forms of function approximation.
For linear and more-general forms of function approximation the parameter vector may
diverge to infinity as in the examples in Chapter 11. As we discussed there, the challengeof o↵-policy learning has two parts. O↵-policy eligibility traces deal e↵ectively with thefirst part of the challenge, correcting for the expected value of the targets, but not at
all with the second part of the challenge, having to do with the distribution of updates.
Algorithmic strategies for meeting the second part of the challenge of o↵-policy learningwith eligibility traces are summarized in Section 12.11.
Exercise 12.13 What are the dutch-trace and replacing-trace versions of o↵-policy
eligibility traces for state-value and action-value methods?
⇤

12.10

Watkins’s Q( ) to Tree-Backup( )Several methods have been proposed over the years to extend Q-learning to eligibility
traces. The original is Watkins’s Q( ), which decays its eligibility traces in the usual way
as long as a greedy action was taken, then cuts the traces to zero after the first non-greedy
action. The backup diagram for Watkins’s Q( ) is shown in Figure 12.12. In Chapter 6,
we unified Q-learning and Expected Sarsa in the o↵-policy version of the latter, which
includes Q-learning as a special case, and generalizes it to arbitrary target policies, andin the previous section of this chapter we completed our treatment of Expected Sarsa bygeneralizing it to o↵-policy eligibility traces. In Chapter 7, however, we distinguishedn-step Expected Sarsa from n-step Tree Backup, where the latter retained the property
of not using importance sampling. It remains then to present the eligibility trace versionof Tree Backup, which we will call Tree-Backup( ), or TB( ) for short. This is arguably
the true successor to Q-learning because it retains its appealing absence of importancesampling even though it can be applied to o↵-policy data.

12.10.

Watkins’s Q( ) to Tree-Backup( )313

Watkins’s Q( )S t AtSt+1 Rt+1
At+1
1

(1

At+2

)

) 2

=1

··

·

···

···

(1

X

St+2 Rt+2

OR···

ST R T
T

St+n Rt+nt 1

First non-greedy actionn 1

Figure 12.12: The backup diagram for Watkins’s Q( ). The series of component updates ends
either with the end of the episode or with the first nongreedy action, whichever comes first.

The concept of TB( ) is straightforward. As shown in its backup diagram in Figure 12.13, the tree-backup updates of each length (from Section 7.5) are weighted in theusual way dependent on the bootstrapping parameter . To get the detailed equations,with the right indices on the general bootstrapping and discounting parameters, it isbest to start with a recursive form (12.20) for the -return using action values, and then
expand the bootstrapping case of the target after the model of (7.16):
✓
hX
i◆
a .
a
Gt = Rt+1 + t+1 (1 t+1 )V̄t (St+1 )+ t+1⇡(a|St+1 )q̂(St+1 , a, wt )+⇡(At+1 |St+1 )Gt+1= Rt+1 +t+1✓

a6=At+1V̄t (St+1 ) +t+1 ⇡(At+1 |St+1 )

⇣

a
Gt+1

q̂(St+1 , At+1 , wt )⌘◆As per the usual pattern, it can also be written approximately (ignoring changes in theapproximate value function) as a sum of TD errors,
Gt a ⇡ q̂(St , At , wt ) +

1
X
k=t

a
k

k
Y

i=t+1

i i ⇡(Ai |Si ),using the expectation form of the action-based TD error (12.28).
Following the same steps as in the previous section, we arrive at a special eligibility
trace update involving the target-policy probabilities of the selected actions,.
zt = t t ⇡(At |St )zt 1 + rq̂(St , At , wt ).

314

Chapter 12: Eligibility Traces

Tree Backup( )S t AtSt+1 Rt+1
At+1
1
St+2 Rt+2

···
(1

At+2

)

···

(1

X

) 2

··

=1

·

1

AT

1

ST R T
T

Figure 12.13: The backup diagram for the

ST

t 1

version of the Tree Backup algorithm.

This, together with the usual parameter-update rule (12.7), defines the TB( ) algorithm.
Like all semi-gradient algorithms, TB( ) is not guaranteed to be stable when used witho↵-policy data and with a powerful function approximator. To obtain those assurances,
TB( ) would have to be combined with one of the methods presented in the next section.
Exercise 12.14 How might Double Expected Sarsa be extended to eligibility traces? ⇤

⇤

12.11

Stable O↵-policy Methods with Traces

Several methods using eligibility traces have been proposed that achieve guarantees
of stability under o↵-policy training, and here we present four of the most importantusing this book’s standard notation, including general bootstrapping and discounting
functions. All are based on either the Gradient-TD or the Emphatic-TD ideas presented
in Sections 11.7 and 11.8. All the algorithms assume linear function approximation,
though extensions to nonlinear function approximation can also be found in the literature.
GTD( ) is the eligibility-trace algorithm analogous to TDC, the better of the twostate-value Gradient-TD prediction algorithms discussed in Section 11.7. Its goal is to.
learn a parameter wt such that v̂(s,w) = wt> x(s) ⇡ v⇡ (s), even from data that is due tofollowing another policy b. Its update is.
wt+1 = wt + ↵ ts zt↵ t+1 (1t+1 )z>t vt xt+1 ,12.11.

with

Stable O↵-policy Methods with Traces315st , zt , and ⇢t defined in the usual ways for state values (12.23) (12.25) (11.1), and.
vt+1 = vt +st zt

vt> xt xt ,(12.30)where, as in Section 11.7, v 2 Rd is a vector of the same dimension as w, initialized tov0 = 0, and > 0 is a second step-size parameter.
GQ( ) is the Gradient-TD algorithm for action values with eligibility traces. Its goal.
is to learn a parameter wt such that q̂(s, a, wt ) = wt> x(s, a) ⇡ q⇡ (s, a) from o↵-policy
data. If the target policy is "-greedy, or otherwise biased toward the greedy policy for q̂,then GQ( ) can be used as a control algorithm. Its update is.
wt+1 = wt + ↵ ta zt↵ t+1 (1t+1 )z>t vt x̄t+1 ,where x̄t is the average feature vector for St under the target policy,
. X
x̄t =
⇡(a|St )x(St , a),aat is the expectation form of the TD error, which can be writtena .
t = Rt+1 +>t+1 wt x̄t+1

wt> xt ,zt is defined in the usual way for action values (12.29), and the rest is as in GTD( ),including the update for vt (12.30).
HTD( ) is a hybrid state-value algorithm combining aspects of GTD( ) and TD( ). Its
most appealing feature is that it is a strict generalization of TD( ) to o↵-policy learning,
meaning that if the behavior policy happens to be the same as the target policy, then
HTD( ) becomes the same as TD( ), which is not true for GTD( ). This is appealing
because TD( ) is often faster than GTD( ) when both algorithms converge, and TD( )requires setting only a single step size. HTD( ) is defined by.
wt+1 = wt + ↵ ts zt + ↵ (zt zbt )> vt (xtt+1 xt+1 ),
⇣ > ⌘
.
.
vt+1 = vt + ts zt
zbt vt (xtwith v0 = 0,
t+1 xt+1 ),.
.
z t = ⇢t t t z t 1 + x t ,with z 1 = 0,.
.
zbt = t t zbt 1 + xt ,with zb 1 = 0,
where > 0 again is a second step-size parameter. In addition to the second set ofweights, vt , HTD( ) also has a second set of eligibility traces, zbt . These are conventional
accumulating eligibility traces for the behavior policy and become equal to zt if all the ⇢t
are 1, which causes the last term in the wt update to be zero and the overall update to
reduce to TD( ).
Emphatic TD( ) is the extension of the one-step Emphatic-TD algorithm (Sections
9.11 and 11.8) to eligibility traces. The resultant algorithm retains strong o↵-policy
convergence guarantees while enabling any degree of bootstrapping, albeit at the cost of316

Chapter 12: Eligibility Traces

high variance and potentially slow convergence. Emphatic TD( ) is defined by.
wt+1 = wt + ↵ t zt.
>
wt> xtt = Rt+1 + t+1 wt xt+1.
.
z t = ⇢t t t z t 1 + M t x t ,with z 1 = 0,.
Mt = t It + (1
t )Ft
.
.
F t = ⇢ t 1 t Ft 1 + I t ,with F0 = i(S0 ),where Mt 0 is the general form of emphasis, Ft 0 is termed the followon trace, and
It 0 is the interest, as described in Section 11.8. Note that Mt , like t , is not really an
additional memory variable. It can be removed from the algorithm by substituting its
definition into the eligibility-trace equation. Pseudocode and software for the true online
version of Emphatic-TD( ) are available on the web (Sutton, 2015b).
In the on-policy case (⇢t = 1, for all t), Emphatic-TD( ) is similar to conventionalTD( ), but still significantly di↵erent. In fact, whereas Emphatic-TD( ) is guaranteed
to converge for all state-dependent functions, TD( ) is not. TD( ) is guaranteed
convergent only for all constant . See Yu’s counterexample (Ghiassian, Rafiee, andSutton, 2016).

12.12

Implementation Issues

It might at first appear that tabular methods using eligibility traces are much more
complex than one-step methods. A naive implementation would require every state (orstate–action pair) to update both its value estimate and its eligibility trace on every time
step. This would not be a problem for implementations on single-instruction, multipledata, parallel computers or in plausible artificial neural network (ANN) implementations,but it is a problem for implementations on conventional serial computers. Fortunately,
for typical values of and the eligibility traces of almost all states are almost always
nearly zero; only those states that have recently been visited will have traces significantly
greater than zero and only these few states need to be updated to closely approximate
these algorithms.
In practice, then, implementations on conventional computers may keep track of andupdate only the few traces that are significantly greater than zero. Using this trick, thecomputational expense of using traces in tabular methods is typically just a few times
that of a one-step method. The exact multiple of course depends on and and on the
expense of the other computations. Note that the tabular case is in some sense the worst
case for the computational complexity of eligibility traces. When function approximation
is used, the computational advantages of not using traces generally decrease. For example,
if ANNs and backpropagation are used, then eligibility traces generally cause only adoubling of the required memory and computation per step. Truncated -return methods(Section 12.3) can be computationally efficient on conventional computers though they
always require some additional memory.

12.13.

Conclusions

12.13

317

Conclusions

Eligibility traces in conjunction with TD errors provide an efficient, incremental way ofshifting and choosing between Monte Carlo and TD methods. The n-step methods ofChapter 7 also enabled this, but eligibility trace methods are more general, often faster tolearn, and o↵er di↵erent computational complexity tradeo↵s. This chapter has o↵ered an
introduction to the elegant, emerging theoretical understanding of eligibility traces for onand o↵-policy learning and for variable bootstrapping and discounting. One aspect of thiselegant theory is true online methods, which exactly reproduce the behavior of expensive
ideal methods while retaining the computational congeniality of conventional TD methods.
Another aspect is the possibility of derivations that automatically convert from intuitive
forward-view methods to more efficient incremental backward-view algorithms. We
illustrated this general idea in a derivation that started with a classical, expensive Monte
Carlo algorithm and ended with a cheap incremental non-TD implementation using thesame novel eligibility trace used in true online TD methods.
As we mentioned in Chapter 5, Monte Carlo methods may have advantages in nonMarkov tasks because they do not bootstrap. Because eligibility traces make TD methods
more like Monte Carlo methods, they also can have advantages in these cases. If one
wants to use TD methods because of their other advantages, but the task is at least
partially non-Markov, then the use of an eligibility trace method is indicated. Eligibility
traces are the first line of defense against both long-delayed rewards and non-Markov
tasks.
By adjusting , we can place eligibility trace methods anywhere along a continuum
from Monte Carlo to one-step TD methods. Where shall we place them? We do not yet
have a good theoretical answer to this question, but a clear empirical answer appears tobe emerging. On tasks with many steps per episode, or many steps within the half-life of
discounting, it appears significantly better to use eligibility traces than not to (e.g., see
Figure 12.14). On the other hand, if the traces are so long as to produce a pure Monte
Carlo method, or nearly so, then performance degrades sharply. An intermediate mixture
appears to be the best choice. Eligibility traces should be used to bring us toward Monte
Carlo methods, but not all the way there. In the future it may be possible to more finely
vary the trade-o↵ between TD and Monte Carlo methods by using variable , but atpresent it is not clear how this can be done reliably and usefully.
Methods using eligibility traces require more computation than one-step methods, but
in return they o↵er significantly faster learning, particularly when rewards are delayed bymany steps. Thus it often makes sense to use eligibility traces when data are scarce andcannot be repeatedly processed, as is often the case in online applications. On the other
hand, in o↵-line applications in which data can be generated cheaply, perhaps from an
inexpensive simulation, then it often does not pay to use eligibility traces. In these cases
the objective is not to get more out of a limited amount of data, but simply to process asmuch data as possible as quickly as possible. In these cases the speedup per datum due totraces is typically not worth their computational cost, and one-step methods are favored.

318

Chapter 12: Eligibility Traces

MOUNTAIN CAR700

RANDOM WALK

0.5

650

accumulating
traces600

Steps perepisode0.4

accumulating
traces550

RMS error
0.3

500
450

replacingtracesreplacingtraces

400
0

240

0.2

0.4

0.6

0.8

1

0

0.2

0.4

0.6

0.8

!

!

PUDDLE WORLDCART AND POLE

0.2

1

300

230
250

220
210

Cost perepisode200

200

replacingtraces190
180

accumulating
tracesFailures per
100,000 steps

150

170

100

160
50

150
0

0.2

0.4

0.6

!

0.8

1

0

0.2

0.4

0.6

0.8

1

!

Figure 12.14: The e↵ect of on reinforcement learning performance in four di↵erent test
problems. In all cases, performance is generally best (a lower number in the graph) at anintermediate value of . The two left panels are applications to simple continuous-state control
tasks using the Sarsa( ) algorithm and tile coding, with either replacing or accumulating traces(Sutton, 1996). The upper-right panel is for policy evaluation on a random walk task using TD( )(Singh and Sutton, 1996). The lower right panel is unpublished data for the pole-balancing task
(Example 3.4) from an earlier study (Sutton, 1984).

12.13.

Conclusions

319

Bibliographical and Historical Remarks
Eligibility traces came into reinforcement learning via the fecund ideas of Klopf (1972).
Our use of eligibility traces is based on Klopf’s work (Sutton, 1978a, 1978b, 1978c; Bartoand Sutton, 1981a, 1981b; Sutton and Barto, 1981a; Barto, Sutton, and Anderson, 1983;
Sutton, 1984). We may have been the first to use the term “eligibility trace” (Suttonand Barto, 1981a). The idea that stimuli produce after e↵ects in the nervous systemthat are important for learning is very old (see Chapter 14). Some of the earliest uses of
eligibility traces were in the actor–critic methods discussed in Chapter 13 (Barto, Sutton,
and Anderson, 1983; Sutton, 1984).
12.1

Compound updates were called “complex backups” in the first edition of thisbook.
The -return and its error-reduction properties were introduced by Watkins (1989)
and further developed by Jaakkola, Jordan, and Singh (1994). The random walk
results in this and subsequent sections are new to this text, as are the terms“forward view” and “backward view.” The notion of a -return algorithm wasintroduced in the first edition of this text. The more refined treatment presented
here was developed in conjunction with Harm van Seijen (e.g., van Seijen andSutton, 2014).

12.2

TD( ) with accumulating traces was introduced by Sutton (1988, 1984). Convergence in the mean was proved by Dayan (1992), and with probability 1 bymany researchers, including Peng (1993), Dayan and Sejnowski (1994), Tsitsiklis(1994), and Gurvits, Lin, and Hanson (1994). The bound on the error of theasymptotic -dependent solution of linear TD( ) is due to Tsitsiklis and Van
Roy (1997).

12.3

Truncated TD methods were developed by Cichosz (1995) and van Seijen (2016).

12.4

The idea of redoing updates was extensively developed by van Seijen, originally
under the name “best-match learning” (van Seijen, 2011; van Seijen, Whiteson,
van Hasselt, and Weiring, 2011).

12.5

True online TD( ) is primarily due to Harm van Seijen (van Seijen and Sutton,
2014; van Seijen et al., 2016) though some of its key ideas were discovered
independently by Hado van Hasselt (personal communication). The name “dutch
traces” is in recognition of the contributions of both scientists. Replacing traces
are due to Singh and Sutton (1996).

12.6

The material in this section is from van Hasselt and Sutton (2015).

12.7

Sarsa( ) with accumulating traces was first explored as a control method by
Rummery and Niranjan (1994; Rummery, 1995). True Online Sarsa( ) was

320

Chapter 12: Eligibility Traces

introduced by van Seijen and Sutton (2014). The algorithm on page 307 wasadapted from van Seijen et al. (2016). The Mountain Car results were made forthis text, except for Figure 12.11 which is adapted from van Seijen and Sutton(2014).
12.8

Perhaps the first published discussion of variable was by Watkins (1989), whopointed out that the cutting o↵ of the update sequence (Figure 12.12) in his
Q( ) when a nongreedy action was selected could be implemented by temporarily
setting to 0.
Variable was introduced in the first edition of this text. The roots of variable
are in the work on options (Sutton, Precup, and Singh, 1999) and its precursors(Sutton, 1995a), becoming explicit in the GQ( ) paper (Maei and Sutton, 2010),
which also introduced some of these recursive forms for the -returns.
A di↵erent notion of variable

12.9

has been developed by Yu (2012).

O↵-policy eligibility traces were introduced by Precup et al. (2000, 2001), thenfurther developed by Bertsekas and Yu (2009), Maei (2011; Maei and Sutton,2010), Yu (2012), and by Sutton, Mahmood, Precup, and van Hasselt (2014).
The last reference in particular gives a powerful forward view for o↵-policy TD
methods with general state-dependent and . The presentation here seems tobe new.
This section ends with an elegant Expected Sarsa( ) algorithm. Although it isa natural algorithm, to our knowledge it has not previously been described ortested in the literature.

12.10 Watkins’s Q( ) is due to Watkins (1989). The tabular, episodic, o↵-line version
has been proven convergent by Munos, Stepleton, Harutyunyan, and Bellemare(2016). Alternative Q( ) algorithms were proposed by Peng and Williams (1994,
1996) and by Sutton, Mahmood, Precup, and van Hasselt (2014). Tree Backup( )is due to Precup, Sutton, and Singh (2000).
12.11 GTD( ) is due to Maei (2011). GQ( ) is due to Maei and Sutton (2010).
HTD( ) is due to White and White (2016) based on the one-step HTD algorithm
introduced by Hackman (2012). The latest developments in the theory ofGradient-TD methods are by Yu (2017). Emphatic TD( ) was introduced by
Sutton, Mahmood, and White (2016), who proved its stability. Yu (2015, 2016)
proved its convergence, and the algorithm was developed further by Hallak etal. (2015, 2016).

Chapter 13

Policy Gradient Methods
In this chapter we consider something new. So far in this book almost all the methods
have been action-value methods; they learned the values of actions and then selectedactions based on their estimated action values1 ; their policies would not even exist without
the action-value estimates. In this chapter we consider methods that instead learn aparameterized policy that can select actions without consulting a value function. A valuefunction may still be used to learn the policy parameter, but is not required for action0
selection. We use the notation ✓ 2 Rd for the policy’s parameter vector. Thus we write⇡(a|s, ✓) = Pr{At = a | St = s, ✓t = ✓} for the probability that action a is taken at time t
given that the environment is in state s at time t with parameter ✓. If a method uses alearned value function as well, then the value function’s weight vector is denoted w 2 Rd
as usual, as in v̂(s,w).
In this chapter we consider methods for learning the policy parameter based on thegradient of some scalar performance measure J(✓) with respect to the policy parameter.
These methods seek to maximize performance, so their updates approximate gradientascent in J:\t ),✓t+1 = ✓t + ↵rJ(✓(13.1)\t ) 2 Rd is a stochastic estimate whose expectation approximates the gradient
where rJ(✓of the performance measure with respect to its argument ✓t . All methods that follow
this general schema we call policy gradient methods, whether or not they also learn anapproximate value function. Methods that learn approximations to both policy and valuefunctions are often called actor–critic methods, where ‘actor’ is a reference to the learned
policy, and ‘critic’ refers to the learned value function, usually a state-value function.
First we treat the episodic case, in which performance is defined as the value of the start
state under the parameterized policy, before going on to consider the continuing case, in
0

1 The lone exception is the gradient bandit algorithms of Section 2.8. In fact, that section goes throughmany of the same steps, in the single-state bandit case, as we go through here for full MDPs. Reviewingthat section would be good preparation for fully understanding this chapter.

322

Chapter 13: Policy Gradient Methods

which performance is defined as the average reward rate, as in Section 10.3. In the end,
we are able to express the algorithms for both cases in very similar terms.

13.1

Policy Approximation and its Advantages

In policy gradient methods, the policy can be parameterized in any way, as long as⇡(a|s, ✓) is di↵erentiable with respect to its parameters, that is, as long as r⇡(a|s, ✓) (thecolumn vector of partial derivatives of ⇡(a|s, ✓) with respect to the components of ✓) exists
0
and is finite for all s 2 S, a 2 A(s), and ✓ 2 Rd . In practice, to ensure exploration we
generally require that the policy never becomes deterministic (i.e., that ⇡(a|s, ✓) 2 (0, 1),
for all s, a, ✓). In this section we introduce the most common parameterization for
discrete action spaces and point out the advantages it o↵ers over action-value methods.
Policy-based methods also o↵er useful ways of dealing with continuous action spaces, as
we describe later in Section 13.7.
If the action space is discrete and not too large, then a natural and common kind ofparameterization is to form parameterized numerical preferences h(s, a, ✓) 2 R for eachstate–action pair. The actions with the highest preferences in each state are given thehighest probabilities of being selected, for example, according to an exponential soft-max
distribution:. eh(s,a,✓)⇡(a|s, ✓) = P h(s,b,✓) ,be(13.2)

where e ⇡ 2.71828 is the base of the natural logarithm. Note that the denominator hereis just what is required so that the action probabilities in each state sum to one. We callthis kind of policy parameterization soft-max in action preferences.
The action preferences themselves can be parameterized arbitrarily. For example, theymight be computed by a deep artificial neural network (ANN), where ✓ is the vectorof all the connection weights of the network (as in the AlphaGo system described in
Section 16.6). Or the preferences could simply be linear in features,h(s, a, ✓) = ✓ > x(s, a),(13.3)
0using feature vectors x(s, a) 2 Rd constructed by any of the methods described in
Section 9.5.
One advantage of parameterizing policies according to the soft-max in action preferencesis that the approximate policy can approach a deterministic policy, whereas with "-greedy
action selection over action values there is always an " probability of selecting a random
action. Of course, one could select according to a soft-max distribution based on action
values, but this alone would not allow the policy to approach a deterministic policy.
Instead, the action-value estimates would converge to their corresponding true values,
which would di↵er by a finite amount, translating to specific probabilities other than 0 and
1. If the soft-max distribution included a temperature parameter, then the temperature
could be reduced over time to approach determinism, but in practice it would be difficult
to choose the reduction schedule, or even the initial temperature, without more priorknowledge of the true action values than we would like to assume. Action preferences

13.1. Policy Approximation and its Advantages

323

are di↵erent because they do not approach specific values; instead they are driven to
produce the optimal stochastic policy. If the optimal policy is deterministic, then thepreferences of the optimal actions will be driven infinitely higher than all suboptimal
actions (if permitted by the parameterization).
A second advantage of parameterizing policies according to the soft-max in action
preferences is that it enables the selection of actions with arbitrary probabilities. In
problems with significant function approximation, the best approximate policy may be
stochastic. For example, in card games with imperfect information the optimal play isoften to do two di↵erent things with specific probabilities, such as when bluffing in Poker.
Action-value methods have no natural way of finding stochastic optimal policies, whereas
policy approximating methods can, as shown in Example 13.1.
Example 13.1 Short corridor with switched actions
Consider the small corridor gridworld shown inset in the graph below. The reward
is 1 per step, as usual. In each of the three nonterminal states there are only
two actions, right and left. These actions have their usual consequences in thefirst and third states (left causes no movement in the first state), but in thesecond state they are reversed, so that right moves to the left and left moves tothe right. The problem is difficult because all the states appear identical under
the function approximation. In particular, we define x(s, right) = [1, 0]> andx(s, left) = [0, 1]> , for all s. An action-value method with "-greedy action selection
is forced to choose between just two policies: choosing right with high probability
1 "/2 on all steps or choosing left with the same high probability on all time
steps. If " = 0.1, then these two policies achieve a value (at the start state)of less than 44 and 82, respectively, as shown in the graph. A method can
do significantly better if it can learn a specific probability with which to selectright. The best probability is about 0.59, which achieves a value of about 11.6.
-11.6

optimalstochastic
policy-20-40-greedy rightJ(✓) = v⇡✓ (S)
-60

S
-80

G

-greedy left

-100
0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

probability of right action

0.8

0.9

1

324

Chapter 13: Policy Gradient Methods

Perhaps the simplest advantage that policy parameterization may have over actionvalue parameterization is that the policy may be a simpler function to approximate.
Problems vary in the complexity of their policies and action-value functions. For some,the action-value function is simpler and thus easier to approximate. For others, the policy
is simpler. In the latter case a policy-based method will typically learn faster and yield asuperior asymptotic policy (as in Tetris; see Şimşek, Algórta, and Kothiyal, 2016).
Finally, we note that the choice of policy parameterization is sometimes a good way
of injecting prior knowledge about the desired form of the policy into the reinforcement
learning system. This is often the most important reason for using a policy-based learning
method.
Exercise 13.1 Use your knowledge of the gridworld and its dynamics to determine anexact symbolic expression for the optimal probability of selecting the right action in
Example 13.1.
⇤

13.2

The Policy Gradient Theorem

In addition to the practical advantages of policy parameterization over "-greedy action
selection, there is also an important theoretical advantage. With continuous policy
parameterization the action probabilities change smoothly as a function of the learned
parameter, whereas in "-greedy selection the action probabilities may change dramaticallyfor an arbitrarily small change in the estimated action values, if that change results in adi↵erent action having the maximal value. Largely because of this, stronger convergenceguarantees are available for policy-gradient methods than for action-value methods. In
particular, it is the continuity of the policy dependence on the parameters that enables
policy-gradient methods to approximate gradient ascent (13.1).
The episodic and continuing cases define the performance measure, J(✓), di↵erently
and thus have to be treated separately to some extent. Nevertheless, we will try topresent both cases uniformly, and we develop a notation so that the major theoreticalresults can be described with a single set of equations.
In this section we treat the episodic case, for which we define the performance measure
as the value of the start state of the episode. We can simplify the notation without
losing any meaningful generality by assuming that every episode starts in some particular(non-random) state s0 . Then, in the episodic case we define performance as.
J(✓) = v⇡✓ (s0 ),(13.4)

where v⇡✓ is the true value function for ⇡✓ , the policy determined by ✓. From here on in
our discussion we will assume no discounting ( = 1) for the episodic case, although forcompleteness we do include the possibility of discounting in the boxed algorithms.
With function approximation it may seem challenging to change the policy parameterin a way that ensures improvement. The problem is that performance depends on boththe action selections and the distribution of states in which those selections are made,and that both of these are a↵ected by the policy parameter. Given a state, the e↵ect ofthe policy parameter on the actions, and thus on reward, can be computed in a relatively
straightforward way from knowledge of the parameterization. But the e↵ect of the policy

13.2. The Policy Gradient Theorem

325

Proof of the Policy Gradient Theorem (episodic case)With just elementary calculus and re-arranging of terms, we can prove the policy
gradient theorem from first principles. To keep the notation simple, we leave it
implicit in all cases that ⇡ is a function of ✓, and all gradients are also implicitlywith respect to ✓. First note that the gradient of the state-value function can be
written in terms of the action-value function as"#Xrv⇡ (s) = r⇡(a|s)q⇡ (s, a) , for all s 2 S(Exercise 3.18)
=

Xh
a

=

Xh
a

=

Xh
a

=

Xh
a

a

r⇡(a|s)q⇡ (s, a) + ⇡(a|s)rq⇡ (s, a)r⇡(a|s)q⇡ (s, a) + ⇡(a|s)rr⇡(a|s)q⇡ (s, a) + ⇡(a|s)
r⇡(a|s)q⇡ (s, a) + ⇡(a|s)

X⇥
a0

=

1
XX

x2S k=0

X
s0 ,r

X
s0

X
s0

Pr(s ! x, k, ⇡)a(product rule of calculus)p(s0 , r |s, a) r + v⇡ (s0 )i(Exercise 3.19 and Equation 3.2)ip(s0 |s, a)rv⇡ (s0 )(Eq. 3.4)p(s0 |s, a)(unrolling)X⇤ir⇡(a0 |s0 )q⇡ (s0 , a0 ) + ⇡(a0 |s0 )
X

i

s00

p(s00 |s0 , a0 )rv⇡ (s00 )r⇡(a|x)q⇡ (x, a),after repeated unrolling, where Pr(s ! x, k, ⇡) is the probability of transitioning
from state s to state x in k steps under policy ⇡. It is then immediate thatrJ(✓) = rv⇡ (s0 )
=

1
X X
s

=

X

k=0

⌘(s)

s

X
a

X
a

r⇡(a|s)q⇡ (s, a)r⇡(a|s)q⇡ (s, a)⌘(s) Xr⇡(a|s)q⇡ (s, a)0s0 ⌘(s ) a
s
s0
X
X
X
=
⌘(s0 )µ(s)r⇡(a|s)q⇡ (s, a)=X

Pr(s0 ! s, k, ⇡)!

⌘(s0 )

s0

/

X
s

X
s

µ(s)

X
a

P

(box page 199)

(Eq. 9.3)ar⇡(a|s)q⇡ (s, a)(Q.E.D.)

326

Chapter 13: Policy Gradient Methods

on the state distribution is a function of the environment and is typically unknown. How
can we estimate the performance gradient with respect to the policy parameter when thegradient depends on the unknown e↵ect of policy changes on the state distribution?
Fortunately, there is an excellent theoretical answer to this challenge in the form of
the policy gradient theorem, which provides an analytic expression for the gradient of
performance with respect to the policy parameter (which is what we need to approximate
for gradient ascent (13.1)) that does not involve the derivative of the state distribution.
The policy gradient theorem for the episodic case establishes thatX
X
rJ(✓) /µ(s)q⇡ (s, a)r⇡(a|s, ✓),(13.5)sawhere the gradients are column vectors of partial derivatives with respect to the components of ✓, and ⇡ denotes the policy corresponding to parameter vector ✓. The symbol /here means “proportional to”. In the episodic case, the constant of proportionality is theaverage length of an episode, and in the continuing case it is 1, so that the relationship is
actually an equality. The distribution µ here (as in Chapters 9 and 10) is the on-policy
distribution under ⇡ (see page 199). The policy gradient theorem is proved for theepisodic case in the box on the previous page.

13.3

REINFORCE: Monte Carlo Policy Gradient

We are now ready to derive our first policy-gradient learning algorithm. Recall our overall
strategy of stochastic gradient ascent (13.1), which requires a way to obtain samples suchthat the expectation of the sample gradient is proportional to the actual gradient of theperformance measure as a function of the parameter. The sample gradients need only be
proportional to the gradient because any constant of proportionality can be absorbed
into the step size ↵, which is otherwise arbitrary. The policy gradient theorem gives anexact expression proportional to the gradient; all that is needed is some way of samplingwhose expectation equals or approximates this expression. Notice that the right-handside of the policy gradient theorem is a sum over states weighted by how often the statesoccur under the target policy ⇡; if ⇡ is followed, then states will be encountered in these
proportions. Thus
X
X
rJ(✓) /µ(s)q⇡ (s, a)r⇡(a|s, ✓)
s

"

= E⇡

a

X
a

#

q⇡ (St , a)r⇡(a|St , ✓) .

(13.6)

We could stop here and instantiate our stochastic gradient-ascent algorithm (13.1) as
X.
✓t+1 = ✓t + ↵q̂(St , a, w)r⇡(a|St , ✓),(13.7)awhere q̂ is some learned approximation to q⇡ . This algorithm, which has been called
an all-actions method because its update involves all of the actions, is promising and13.3. REINFORCE: Monte Carlo Policy Gradient327

deserving of further study, but our current interest is the classical REINFORCE algorithm(Willams, 1992) whose update at time t involves just At , the one action actually taken attime t.
We continue our derivation of REINFORCE by introducing At in the same way as we
introduced St in (13.6)—by replacing a sum over the random variable’s possible valuesby an expectation under ⇡, and then sampling the expectation. Equation (13.6) involves
an appropriate sum over actions, but each term is not weighted by ⇡(a|St , ✓) as is needed
for an expectation under ⇡. So we introduce such a weighting, without changing theequality, by multiplying and then dividing the summed terms by ⇡(a|St , ✓). Continuingfrom (13.6), we have"#Xr⇡(a|St , ✓)rJ(✓) / E⇡⇡(a|St , ✓)q⇡ (St , a)⇡(a|St , ✓)ar⇡(At |St , ✓)= E⇡ q⇡ (St , At )(replacing a by the sample At ⇠ ⇡)⇡(At |St , ✓)r⇡(At |St , ✓)= E ⇡ Gt,
(because E⇡[Gt |St , At ] = q⇡ (St , At ))⇡(At |St , ✓)where Gt is the return as usual. The final expression in brackets is exactly what is needed,
a quantity that can be sampled on each time step whose expectation is proportional
to the gradient. Using this sample to instantiate our generic stochastic gradient ascentalgorithm (13.1) yields the REINFORCE update:r⇡(At |St , ✓t ).
✓t+1 = ✓t + ↵Gt.
⇡(At |St , ✓t )(13.8)

This update has an intuitive appeal. Each increment is proportional to the product of a
return Gt and a vector, the gradient of the probability of taking the action actually taken
divided by the probability of taking that action. The vector is the direction in parameter
space that most increases the probability of repeating the action At on future visits
to state St . The update increases the parameter vector in this direction proportional
to the return, and inversely proportional to the action probability. The former makes
sense because it causes the parameter to move most in the directions that favor actions
that yield the highest return. The latter makes sense because otherwise actions that are
selected frequently are at an advantage (the updates will be more often in their direction)and might win out even if they do not yield the highest return.
Note that REINFORCE uses the complete return from time t, which includes all
future rewards up until the end of the episode. In this sense REINFORCE is a Monte
Carlo algorithm and is well defined only for the episodic case with all updates made in
retrospect after the episode is completed (like the Monte Carlo algorithms in Chapter 5).
This is shown explicitly in the boxed algorithm on the next page.
Notice that the update in the last line of pseudocode appears rather di↵erent fromthe REINFORCE update rule (13.8). One di↵erence is that the pseudocode uses thet |St ,✓t )compact expression r ln ⇡(At |St , ✓t ) for the fractional vector r⇡(A⇡(At |St ,✓t ) in (13.8). That
these two expressions for the vector are equivalent follows from the identity r ln x = rx
x .

328

Chapter 13: Policy Gradient Methods

This vector has been given several names and notations in the literature; we will refer
to it simply as the eligibility vector. Note that it is the only place that the policy
parameterization appears in the algorithm.
REINFORCE: Monte-Carlo Policy-Gradient Control (episodic) for ⇡⇤Input: a di↵erentiable policy parameterization ⇡(a|s, ✓)Algorithm parameter: step size ↵ > 0
0Initialize policy parameter ✓ 2 Rd (e.g., to 0)
Loop forever (for each episode):
Generate an episode S0 , A0 , R1 , . . . , ST 1 , AT 1 , RT , following ⇡(·|·, ✓)Loop for each step of the episode t = 0, 1, . . . , T 1:
PT
k t 1
G
Rk
k=t+1
t
✓
✓ + ↵ Gr ln ⇡(At |St , ✓)(Gt )The second di↵erence between the pseudocode update and the REINFORCE update
equation (13.8) is that the former includes a factor of t . This is because, as mentioned
earlier, in the text we are treating the non-discounted case ( = 1) while in the boxedalgorithms we are giving the algorithms for the general discounted case. All of the ideas
go through in the discounted case with appropriate adjustments (including to the box onpage 199) but involve additional complexity that distracts from the main ideas.
⇤

Exercise 13.2 Generalize the box on page 199, the policy gradient theorem (13.5), theproof of the policy gradient theorem (page 325), and the steps leading to the REINFORCE
update equation (13.8), so that (13.8) ends up with a factor of t and thus aligns withthe general algorithm given in the pseudocode.
⇤
Figure 13.1 shows the performance of REINFORCE on the short-corridor gridworldfrom Example 13.1.
v⇤ (s0 )

-10
-20

G0

↵ = 2 13

↵ = 2 14↵ = 2 12-40

Total reward
on episodeaveraged over 100 runs

-60

-80
-90
1

200

400

600

800

1000

EpisodeFigure 13.1: REINFORCE on the short-corridor gridworld (Example 13.1). With a good stepsize, the total reward per episode approaches the optimal value of the start state.

13.4. REINFORCE with Baseline329

As a stochastic gradient method, REINFORCE has good theoretical convergenceproperties. By construction, the expected update over an episode is in the same direction
as the performance gradient. This assures an improvement in expected performance forsufficiently small ↵, and convergence to a local optimum under standard stochastic approximation conditions for decreasing ↵. However, as a Monte Carlo method REINFORCE
may be of high variance and thus produce slow learning.
Exercise 13.3 In Section 13.1 we considered policy parameterizations using the soft-max inaction preferences (13.2) with linear action preferences (13.3). For this parameterization,
prove that the eligibility vector isX
r ln ⇡(a|s, ✓) = x(s, a)⇡(b|s, ✓)x(s, b),(13.9)busing the definitions and elementary calculus.

13.4

⇤

REINFORCE with Baseline

The policy gradient theorem (13.5) can be generalized to include a comparison of theaction value to an arbitrary baseline b(s):⌘X
X⇣
rJ(✓) /µ(s)q⇡ (s, a) b(s) r⇡(a|s, ✓).
(13.10)sa

The baseline can be any function, even a random variable, as long as it does not varywith a; the equation remains valid because the subtracted quantity is zero:XXb(s)r⇡(a|s, ✓) = b(s)r⇡(a|s, ✓) = b(s)r1 = 0.
aaThe policy gradient theorem with baseline (13.10) can be used to derive an update
rule using similar steps as in the previous section. The update rule that we end up withis a new version of REINFORCE that includes a general baseline:⇣⌘ r⇡(A |S , ✓ ).
t tt✓t+1 = ✓t + ↵ Gt b(St ).
(13.11)⇡(At |St , ✓t )Because the baseline could be uniformly zero, this update is a strict generalization of
REINFORCE. In general, the baseline leaves the expected value of the update unchanged,
but it can have a large e↵ect on its variance. For example, we saw in Section 2.8 that an
analogous baseline can significantly reduce the variance (and thus speed the learning) of
gradient bandit algorithms. In the bandit algorithms the baseline was just a number (theaverage of the rewards seen so far), but for MDPs the baseline should vary with state.
In some states all actions have high values and we need a high baseline to di↵erentiate
the higher valued actions from the less highly valued ones; in other states all actions will
have low values and a low baseline is appropriate.
One natural choice for the baseline is an estimate of the state value, v̂(St ,w), wherew 2 Rd is a weight vector learned by one of the methods presented in previous chapters.

330

Chapter 13: Policy Gradient Methods

Because REINFORCE is a Monte Carlo method for learning the policy parameter, ✓,it seems natural to also use a Monte Carlo method to learn the state-value weights, w.
A complete pseudocode algorithm for REINFORCE with baseline using such a learned
state-value function as the baseline is given in the box below.
REINFORCE with Baseline (episodic), for estimating ⇡✓ ⇡ ⇡⇤Input: a di↵erentiable policy parameterization ⇡(a|s, ✓)Input: a di↵erentiable state-value function parameterization v̂(s,w)
Algorithm parameters: step sizes ↵✓ > 0, ↵w > 00
Initialize policy parameter ✓ 2 Rd and state-value weights w 2 Rd (e.g., to 0)
Loop forever (for each episode):
Generate an episode S0 , A0 , R1 , . . . , ST 1 , AT 1 , RT , following ⇡(·|·, ✓)Loop for each step of the episode t = 0, 1, . . . , T 1:
PT
k t 1
G
Rk
k=t+1
G v̂(St ,w)w
w + ↵w rv̂(St ,w)✓✓ + ↵✓ t r ln ⇡(At |St , ✓)(Gt )This algorithm has two step sizes, denoted ↵✓ and ↵w (where ↵✓ is the ↵ in (13.11)).
Choosing the step size for values (here ↵w ) is relatively
easy; in⇤ the linear case we have⇥2
rules of thumb for setting it, such as ↵w = 0.1/E krv̂(St ,w)kµ (see Section 9.6). It ismuch less clear how to set the step size for the policy parameters, ↵✓ , whose best value
depends on the range of variation of the rewards and on the policy parameterization.
REINFORCE with baseline

-10

↵=2

-20

↵✓ = 2 9 , ↵w = 2 6v⇤ (s0 )9

REINFORCE

G0

Total reward
on episodeaveraged over 100 runs

↵ = 2 13

-40

-60

-80
-90
1

200

400

600

800

1000

EpisodeFigure 13.2: Adding a baseline to REINFORCE can make it learn much faster, as illustrated here on the short-corridor gridworld (Example 13.1). The step size used here for plain
REINFORCE is that at which it performs best (to the nearest power of two; see Figure 13.1).

13.5. Actor–Critic Methods

331

Figure 13.2 compares the behavior of REINFORCE with and without a baseline on
the short-corridor gridword (Example 13.1). Here the approximate state-value functionused in the baseline is v̂(s,w) = w. That is, w is a single component, w.

13.5

Actor–Critic Methods

In REINFORCE with baseline, the learned state-value function estimates the value ofthe first state of each state transition. This estimate sets a baseline for the subsequent
return, but is made prior to the transition’s action and thus cannot be used to assess that
action. In actor–critic methods, on the other hand, the state-value function is appliedalso to the second state of the transition. The estimated value of the second state, when
discounted and added to the reward, constitutes the one-step return, Gt:t+1 , which is a
useful estimate of the actual return and thus is a way of assessing the action. As we haveseen in the TD learning of value functions throughout this book, the one-step return isoften superior to the actual return in terms of its variance and computational congeniality,
even though it introduces bias. We also know how we can flexibly modulate the extent
of the bias with n-step returns and eligibility traces (Chapters 7 and 12). When thestate-value function is used to assess actions in this way it is called a critic, and theoverall policy-gradient method is termed an actor–critic method. Note that the bias in
the gradient estimate is not due to bootstrapping as such; the actor would be biased even
if the critic was learned by a Monte Carlo method.
First consider one-step actor–critic methods, the analog of the TD methods introducedin Chapter 6 such as TD(0), Sarsa(0), and Q-learning. The main appeal of one-stepmethods is that they are fully online and incremental, yet avoid the complexities of
eligibility traces. They are a special case of the eligibility trace methods, but easierto understand. One-step actor–critic methods replace the full return of REINFORCE(13.11) with the one-step return (and use a learned state-value function as the baseline)
as follows:⇣.
✓t+1 = ✓t + ↵ Gt:t+1v̂(St ,w)

⌘ r⇡(A |S , ✓ )ttt⇡(At |St , ✓t )⇣⌘ r⇡(A |S , ✓ )t tt= ✓t + ↵ Rt+1 + v̂(St+1 ,w) v̂(St ,w)⇡(At |St , ✓t )r⇡(At |St , ✓t )= ✓t + ↵ t.
⇡(At |St , ✓t )(13.12)(13.13)(13.14)

The natural state-value-function learning method to pair with this is semi-gradient TD(0).
Pseudocode for the complete algorithm is given in the box at the top of the next page.
Note that it is now a fully online, incremental algorithm, with states, actions, and rewards
processed as they occur and then never revisited.

332

Chapter 13: Policy Gradient Methods

One-step Actor–Critic (episodic), for estimating ⇡✓ ⇡ ⇡⇤Input: a di↵erentiable policy parameterization ⇡(a|s, ✓)Input: a di↵erentiable state-value function parameterization v̂(s,w)
Parameters: step sizes ↵✓ > 0, ↵w > 00
Initialize policy parameter ✓ 2 Rd and state-value weights w 2 Rd (e.g., to 0)
Loop forever (for each episode):
Initialize S (first state of episode)I
1
Loop while S is not terminal (for each time step):A ⇠ ⇡(·|S, ✓)Take action A, observe S 0 , R.
R + v̂(S 0 ,w) v̂(S,w)(if S 0 is terminal, then v̂(S 0 ,w) = 0)w
w
w + ↵ rv̂(S,w)✓✓ + ↵✓ I r ln ⇡(A|S, ✓)I
I
S
S0The generalizations to the forward view of n-step methods and then to a -returnalgorithm are straightforward. The one-step return in (13.12) is merely replaced by Gt:t+n
or Gt respectively. The backward view of the -return algorithm is also straightforward,
using separate eligibility traces for the actor and critic, each after the patterns in
Chapter 12. Pseudocode for the complete algorithm is given in the box below.
Actor–Critic with Eligibility Traces (episodic), for estimating ⇡✓ ⇡ ⇡⇤Input: a di↵erentiable policy parameterization ⇡(a|s, ✓)Input: a di↵erentiable state-value function parameterization v̂(s,w)
Parameters: trace-decay rates ✓ 2 [0, 1], w 2 [0, 1]; step sizes ↵✓ > 0, ↵w > 00
Initialize policy parameter ✓ 2 Rd and state-value weights w 2 Rd (e.g., to 0)
Loop forever (for each episode):
Initialize S (first state of episode)z✓
0 (d0 -component eligibility trace vector)wz
0 (d-component eligibility trace vector)
I
1
Loop while S is not terminal (for each time step):A ⇠ ⇡(·|S, ✓)Take action A, observe S 0 , R.
R + v̂(S 0 ,w) v̂(S,w)(if S 0 is terminal, then v̂(S 0 ,w) = 0)w w
w
zz + rv̂(S,w)✓ ✓z✓z + I r ln ⇡(A|S, ✓)w
w + ↵ w zw✓
✓ + ↵ ✓ z✓
I
I
S
S0

13.6. Policy Gradient for Continuing Problems

13.6

333

Policy Gradient for Continuing Problems

As discussed in Section 10.3, for continuing problems without episode boundaries we needto define performance in terms of the average rate of reward per time step:
h

1X
.
.
J(✓) = r(⇡) = lim
E[Rt | S0 , A0:t 1 ⇠ ⇡]
h!1 h
t=1

(13.15)= lim E[Rt | S0 , A0:t 1 ⇠ ⇡]
t!1
X
X
X
=
µ(s)
⇡(a|s)p(s0 , r |s, a)r,
s

a

s0,r

.
where µ is the steady-state distribution under ⇡, µ(s) = limt!1 Pr{St = s|A0:t ⇠ ⇡},which is assumed to exist and to be independent of S0 (an ergodicity assumption).
Remember that this is the special distribution under which, if you select actions according
to ⇡, you remain in the same distribution:
X
s

µ(s)

X
a

⇡(a|s, ✓)p(s0 |s, a) = µ(s0 ), for all s0 2 S.

(13.16)

Complete pseudocode for the actor–critic algorithm in the continuing case (backwardview) is given in the box below.
Actor–Critic with Eligibility Traces (continuing), for estimating ⇡✓ ⇡ ⇡⇤Input: a di↵erentiable policy parameterization ⇡(a|s, ✓)Input: a di↵erentiable state-value function parameterization v̂(s,w)
Algorithm parameters: w 2 [0, 1], ✓ 2 [0, 1], ↵w > 0, ↵✓ > 0, ↵R̄ > 0
Initialize R̄ 2 R (e.g., to 0)0
Initialize state-value weights w 2 Rd and policy parameter ✓ 2 Rd (e.g., to 0)
Initialize S 2 S (e.g., to s0 )zw
0 (d-component eligibility trace vector)z✓
0 (d0 -component eligibility trace vector)
Loop forever (for each time step):A ⇠ ⇡(·|S, ✓)Take action A, observe S 0 , R
R R̄ + v̂(S 0 ,w) v̂(S,w)
R̄
R̄ + ↵R̄w w
w
zz + rv̂(S,w)✓ ✓✓zz + r ln ⇡(A|S, ✓)w
w + ↵ w zw✓
✓ + ↵ ✓ z✓
S
S0

334

Chapter 13: Policy Gradient Methods.
.
Naturally, in the continuing case, we define values, v⇡ (s) = E⇡[Gt |St = s] and q⇡ (s, a) =E⇡[Gt |St = s, At = a], with respect to the di↵erential return:.
Gt = Rt+1

r(⇡) + Rt+2r(⇡) + Rt+3r(⇡) + · · · .

(13.17)

With these alternate definitions, the policy gradient theorem as given for the episodic
case (13.5) remains true for the continuing case. A proof is given in the box on the next
page. The forward and backward view equations also remain the same.
Proof of the Policy Gradient Theorem (continuing case)
The proof of the policy gradient theorem for the continuing case begins similarly
to the episodic case. Again we leave it implicit in all cases that ⇡ is a functionof ✓ and that the gradients are with respect to ✓. Recall that in the continuingcase J(✓) = r(⇡) (13.15) and that v⇡ and q⇡ denote values with respect to thedi↵erential return (13.17). The gradient of the state-value function can be written,
for any s 2 S, as"#X
rv⇡ (s) = r⇡(a|s)q⇡ (s, a) , for all s 2 S(Exercise 3.18)
=

Xh
a

=

Xh
a

=

Xh
a

a

r⇡(a|s)q⇡ (s, a) + ⇡(a|s)rq⇡ (s, a)r⇡(a|s)q⇡ (s, a) + ⇡(a|s)rr⇡(a|s)q⇡ (s, a) + ⇡(a|s)

⇥

X
s0 ,r

i

(product rule of calculus)p(s0 , r |s, a) rrr(✓) +

Xs0r(✓) + v⇡ (s0 )⇤ip(s0 |s, a)rv⇡ (s0 ) .

After re-arranging terms, we obtaini
XhX
rr(✓) =
r⇡(a|s)q⇡ (s, a) + ⇡(a|s)
p(s0 |s, a)rv⇡ (s0 )
s0

a

i

rv⇡ (s).

Notice that the left-hand side can be written rJ(✓), and that it does not depend
on s. Thus the right-hand side does not depend on s either, and we
Pcan safely sum
it over all s 2 S, weighted by µ(s), without changing it (because s µ(s) = 1):
!
i
X
Xh
X
rJ(✓) =µ(s)r⇡(a|s)q⇡ (s, a) + ⇡(a|s)
p(s0 |s, a)rv⇡ (s0 )rv⇡ (s)
s

=

X

s0

a

µ(s)

s

+

X
s

X
a

r⇡(a|s)q⇡ (s, a)

µ(s)

X
a

⇡(a|s)X
s0

p(s0 |s, a)rv⇡ (s0 )X
sµ(s)rv⇡ (s)13.7. Policy Parameterization for Continuous Actions

=

X

µ(s)

s

a

+

XX
s0

=

X

µ(s)

s

=

X
s

13.7

X

|

a

µ(s)

r⇡(a|s)q⇡ (s, a)
µ(s)

s

X
X
a

335

X
a

⇡(a|s)p(s0 |s, a) rv⇡ (s0 ){zµ(s0 ) (13.16)

r⇡(a|s)q⇡ (s, a) +r⇡(a|s)q⇡ (s, a).

X

}

X

µ(s0 )rv⇡ (s0 )s0µ(s)rv⇡ (s)sXµ(s)rv⇡ (s)s

Q.E.D.

Policy Parameterization for Continuous Actions

Policy-based methods o↵er practical ways of dealing with large action spaces, evencontinuous spaces with an infinite number of actions. Instead of computing learned
probabilities for each of the many actions, we instead learn statistics of the probability
distribution. For example, the action set might be the real numbers, with actions chosenfrom a normal (Gaussian) distribution.
The probability density function for the normal distribution is conventionally written✓◆
1(x µ)2.
p(x) = p exp,(13.18)2 2
2⇡where µ and here are the mean and standard deviation of the normal distribution, 1.0µ = 0, σ = 0.2,
and of course ⇡ here is just the numberµ = 0, σ = 1.0,
µ = 0, σ = 5.0,
⇡ ⇡ 3.14159. The probability density func- 0.8◆✓µ = −2, σ = 0.5,
2
1
(x
µ)
tions for several di↵erent means andstan.
p(x) = p exp0.6
2 2
2⇡ ◆dard deviations are shown to the right.
The✓
1 the probability(x µ)2. ofvalue p(x) is the density
0.4
p(x) = p exp
2 2
at x, not the probability. It2⇡can be greaterthan 1; it is the total area under p(x) that 0.2
must sum to 1. In general, one can takethe integral under p(x) for any range of x 0.0
−5
−3
1
3
5
−4
−2
−1
0
2
4
values to get the probability of x fallingx
within that range.
To produce a policy parameterization, the policy can be defined as the normal probability density over a real-valued scalar action, with mean and standard deviation givenby parametric function approximators that depend on the state. That is,✓◆1(a µ(s, ✓))2
.
p exp
,
(13.19)⇡(a|s, ✓) =2 (s, ✓)2(s, ✓) 2⇡2222φµ,σ (x)20where µ : S⇥Rd ! R and0: S⇥Rd ! R+ are two parameterized function approximators.

336

Chapter 13: Policy Gradient Methods

To complete the example we need only give a form for these approximators. For this we
divide the policy’s parameter vector into two parts, ✓ = [✓µ , ✓ ]> , one part to be used
for the approximation of the mean and one part for the approximation of the standard
deviation. The mean can be approximated as a linear function. The standard deviationmust always be positive and is better approximated as the exponential of a linear function.
Thus
⇣
⌘
.
.
µ(s, ✓) = ✓µ > xµ (s) and(s, ✓) = exp ✓ > x (s) ,(13.20)where xµ (s) and x (s) are state feature vectors perhaps constructed by one of the methods
described in Section 9.5. With these definitions, all the algorithms described in the restof this chapter can be applied to learn to select real-valued actions.
Exercise 13.4 Show that for the Gaussian policy parameterization (Equations 13.19 and13.20) the eligibility vector has the following two parts:
r ln ⇡(a|s, ✓µ ) =r⇡(a|s, ✓µ )=⇡(a|s, ✓)r⇡(a|s, ✓ )r ln ⇡(a|s, ✓ ) ==⇡(a|s, ✓)1a(s, ✓)2aµ(s, ✓) xµ (s), andµ(s, ✓)(s, ✓)2

2!

1 x (s).

⇤Exercise 13.5 A Bernoulli-logistic unit is a stochastic neuron-like unit used in some ANNs
(Section 9.7). Its input at time t is a feature vector x(St ); its output, At , is a randomvariable having two values, 0 and 1, with Pr{At = 1} = Pt and Pr{At = 0} = 1 Pt (theBernoulli distribution). Let h(s, 0, ✓) and h(s, 1, ✓) be the preferences in state s for theunit’s two actions given policy parameter ✓. Assume that the di↵erence between theaction preferences is given by a weighted sum of the unit’s input vector, that is, assume
that h(s, 1, ✓) h(s, 0, ✓) = ✓ > x(s), where ✓ is the unit’s weight vector.
(a) Show that if the exponential soft-max distribution (13.2) is used to convert action
preferences to policies, then Pt = ⇡(1|St , ✓t ) = 1/(1 + exp( ✓t> x(St ))) (the logistic
function).
(b) What is the Monte-Carlo REINFORCE update of ✓t to ✓t+1 upon receipt of returnGt ?
(c) Express the eligibility r ln ⇡(a|s, ✓) for a Bernoulli-logistic unit, in terms of a, x(s),and ⇡(a|s, ✓) by calculating the gradient.
Hint for part (c): Define P = ⇡(1|s, ✓) and compute the derivative of the logarithm, foreach action, using the chain rule on P . Combine the two results into one expression that
depends on a and P , and then use the chain rule again, this time on ✓ > x(s), noting thatthe derivative of the logistic function f (x) = 1/(1 + e x ) is f (x)(1 f (x)).
⇤

13.8. Summary

13.8

337

Summary

Prior to this chapter, this book focused on action-value methods—meaning methods thatlearn action values and then use them to determine action selections. In this chapter, on
the other hand, we considered methods that learn a parameterized policy that enables
actions to be taken without consulting action-value estimates. In particular, we haveconsidered policy-gradient methods—meaning methods that update the policy parameteron each step in the direction of an estimate of the gradient of performance with respect
to the policy parameter.
Methods that learn and store a policy parameter have many advantages. They canlearn specific probabilities for taking the actions. They can learn appropriate levelsof exploration and approach deterministic policies asymptotically. They can naturally
handle continuous action spaces. All these things are easy for policy-based methods but
awkward or impossible for "-greedy methods and for action-value methods in general. In
addition, on some problems the policy is just simpler to represent parametrically thanthe value function; these problems are more suited to parameterized policy methods.
Parameterized policy methods also have an important theoretical advantage overaction-value methods in the form of the policy gradient theorem, which gives an exact
formula for how performance is a↵ected by the policy parameter that does not involve
derivatives of the state distribution. This theorem provides a theoretical foundation for
all policy gradient methods.
The REINFORCE method follows directly from the policy gradient theorem. Adding
a state-value function as a baseline reduces REINFORCE’s variance without introducing
bias. If the state-value function is also used to assess—or criticize—the policy’s action
selections, then the value function is called a critic and the policy is called an actor ;the overall method is called an actor–critic method. The critic introduces bias into theactor’s gradient estimates, but is often desirable for the same reason that bootstrapping
TD methods are often superior to Monte Carlo methods (substantially reduced variance).
Overall, policy-gradient methods provide a significantly di↵erent set of strengths andweaknesses than action-value methods. Today they are less well understood in somerespects, but a subject of excitement and ongoing research.

Bibliographical and Historical Remarks
Methods that we now see as related to policy gradients were actually some of the earliest
to be studied in reinforcement learning (Witten, 1977; Barto, Sutton, and Anderson,
1983; Sutton, 1984; Williams, 1987, 1992) and in predecessor fields (see Phansalkar
and Thathachar, 1995). They were largely supplanted in the 1990s by the action-value
methods that are the focus of the other chapters of this book. In recent years, however,attention has returned to actor–critic methods and to policy-gradient methods in general.
Among the further developments beyond what we cover here are natural-gradient methods(Amari, 1998; Kakade, 2002, Peters, Vijayakumar and Schaal, 2005; Peters and Schaal,
2008; Park, Kim and Kang, 2005; Bhatnagar, Sutton, Ghavamzadeh and Lee, 2009; see
Grondman, Busoniu, Lopes and Babuska, 2012), deterministic policy-gradient methods

338

Chapter 13: Policy Gradient Methods(Silver et al., 2014), o↵-policy policy-gradient methods (Degris, White, and Sutton, 2012;
Maei, 2018), and entropy regularization (see Schulman, Chen, and Abbeel, 2017). Major
applications include acrobatic helicopter autopilots and AlphaGo (Section 16.6).
Our presentation in this chapter is based primarily on that by Sutton, McAllester,Singh, and Mansour (2000), who introduced the term “policy gradient methods.” A
useful overview is provided by Bhatnagar et al. (2009). One of the earliest related works
is by Aleksandrov, Sysoyev, and Shemeneva (1968). Thomas (2014) first realized thatthe factor of t , as specified in the boxed algorithms of this chapter, was needed in thecase of discounted episodic problems.
13.1

Example 13.1 and the results with it in this chapter were developed with Eric
Graves.

13.2

The policy gradient theorem here and on page 334 was first obtained by Marbach
and Tsitsiklis (1998, 2001) and then independently by Sutton et al. (2000). A
similar expression was obtained by Cao and Chen (1997). Other early results
are due to Konda and Tsitsiklis (2000, 2003), Baxter and Bartlett (2001), andBaxter, Bartlett, and Weaver (2001). Some additional results are developed by
Sutton, Singh, and McAllester (2000).

13.3

REINFORCE is due to Williams (1987, 1992). Phansalkar and Thathachar(1995) proved both local and global convergence theorems for modified versionsof REINFORCE algorithms.
The all-actions algorithm was first presented in an unpublished but widely
circulated incomplete paper (Sutton, Singh, and McAllester, 2000) and thendeveloped further by Ciosek and Whiteson (2017, 2018), who termed it “expected
policy gradients,” and by Asadi, Allen, Roderick, Mohamed, Konidaris, andLittman (2017), who called it “mean actor critic.”

13.4

The baseline was introduced in Williams’s (1987, 1992) original work. Greensmith,Bartlett, and Baxter (2004) analyzed an arguably better baseline (see Dick, 2015).
Thomas and Brunskill (2017) argue that an action-dependent baseline can beused without incurring bias.

13.5–6 Actor–critic methods were among the earliest to be investigated in reinforcement
learning (Witten, 1977; Barto, Sutton, and Anderson, 1983; Sutton, 1984). The
algorithms presented here are based on the work of Degris, White, and Sutton(2012). Actor–critic methods are sometimes referred to as advantage actor–critic(“A2C”) methods in the literature.
13.7

The first to show how continuous actions could be handled this way appears
to have been Williams (1987, 1992). The figure on page 335 is adapted from
Wikipedia.

Part III: Looking Deeper

In this last part of the book we look beyond the standard reinforcement learning ideas
presented in the first two parts of the book to briefly survey their relationships withpsychology and neuroscience, a sampling of reinforcement learning applications, and someof the active frontiers for future reinforcement learning research.

Chapter 14

Psychology
In previous chapters we developed ideas for algorithms based on computational considerations alone. In this chapter we look at some of these algorithms from anotherperspective: the perspective of psychology and its study of how animals learn. The goalsof this chapter are, first, to discuss ways that reinforcement learning ideas and algorithms
correspond to what psychologists have discovered about animal learning, and second, toexplain the influence reinforcement learning is having on the study of animal learning.
The clear formalism provided by reinforcement learning that systemizes tasks, returns,
and algorithms is proving to be enormously useful in making sense of experimental data,in suggesting new kinds of experiments, and in pointing to factors that may be critical tomanipulate and to measure. The idea of optimizing return over the long term that is
at the core of reinforcement learning is contributing to our understanding of otherwise
puzzling features of animal learning and behavior.
Some of the correspondences between reinforcement learning and psychological theories
are not surprising because the development of reinforcement learning drew inspirationfrom psychological learning theories. However, as developed in this book, reinforcement
learning explores idealized situations from the perspective of an artificial intelligence
researcher or engineer, with the goal of solving computational problems with efficientalgorithms, rather than to replicate or explain in detail how animals learn. As a result,some of the correspondences we describe connect ideas that arose independently in theirrespective fields. We believe these points of contact are specially meaningful because they
expose computational principles important to learning, whether it is learning by artificial
or by natural systems.
For the most part, we describe correspondences between reinforcement learning andlearning theories developed to explain how animals like rats, pigeons, and rabbits learn
in controlled laboratory experiments. Thousands of these experiments were conducted
throughout the 20th century, and many are still being conducted today. Althoughsometimes dismissed as irrelevant to wider issues in psychology, these experiments probe
subtle properties of animal learning, often motivated by precise theoretical questions.
As psychology shifted its focus to more cognitive aspects of behavior, that is, to mental

342

Chapter 14: Psychology

processes such as thought and reasoning, animal learning experiments came to playless of a role in psychology than they once did. But this experimentation led to thediscovery of learning principles that are elemental and widespread throughout the animal
kingdom, principles that should not be neglected in designing artificial learning systems.
In addition, as we shall see, some aspects of cognitive processing connect naturally to thecomputational perspective provided by reinforcement learning.
This chapter’s final section includes references relevant to the connections we discuss aswell as to connections we neglect. We hope this chapter encourages readers to probe allof these connections more deeply. Also included in this final section is a discussion of how
the terminology used in reinforcement learning relates to that of psychology. Many of
the terms and phrases used in reinforcement learning are borrowed from animal learningtheories, but the computational/engineering meanings of these terms and phrases do notalways coincide with their meanings in psychology.

14.1

Prediction and Control

The algorithms we describe in this book fall into two broad categories: algorithms
for prediction and algorithms for control.1 These categories arise naturally in solution
methods for the reinforcement learning problem presented in Chapter 3. In many ways
these categories respectively correspond to categories of learning extensively studied
by psychologists: classical, or Pavlovian, conditioning and instrumental, or operant,
conditioning. These correspondences are not completely accidental because of psychology’s
influence on reinforcement learning, but they are nevertheless striking because they connect
ideas arising from di↵erent objectives.
The prediction algorithms presented in this book estimate quantities that dependon how features of an agent’s environment are expected to unfold over the future. We
specifically focus on estimating the amount of reward an agent can expect to receive overthe future while it interacts with its environment. In this role, prediction algorithms arepolicy evaluation algorithms, which are integral components of algorithms for improving
policies. But prediction algorithms are not limited to predicting future reward; they can
predict any feature of the environment (see, for example, Modayil, White, and Sutton,2014). The correspondence between prediction algorithms and classical conditioning restson their common property of predicting upcoming stimuli, whether or not those stimuli
are rewarding (or punishing).
The situation in an instrumental, or operant, conditioning experiment is di↵erent.
Here, the experimental apparatus is set up so that an animal is given something it likes(a reward) or something it dislikes (a penalty) depending on what the animal did. The
animal learns to increase its tendency to produce rewarded behavior and to decrease its
tendency to produce penalized behavior. The reinforcing stimulus is said to be contingent
on the animal’s behavior, whereas in classical conditioning it is not (although it is difficult
to remove all behavior contingencies in a classical conditioning experiment). Instrumental
1 What control means for us is di↵erent from what it typically means in animal learning theories; there
the environment controls the agent instead of the other way around. See our comments on terminology
at the end of this chapter.

14.2. Classical Conditioning343

conditioning experiments are like those that inspired Thorndike’s Law of E↵ect that
we briefly discuss in Chapter 1. Control is at the core of this form of learning, whichcorresponds to the operation of reinforcement learning’s policy-improvement algorithms.
Thinking of classical conditioning in terms of prediction, and instrumental conditioningin terms of control, is a starting point for connecting our computational view of reinforcement learning to animal learning, but in reality, the situation is more complicatedthan this. There is more to classical conditioning than prediction; it also involves action,and so is a mode of control, sometimes called Pavlovian control. Further, classical andinstrumental conditioning interact in interesting ways, with both sorts of learning likely
being engaged in most experimental situations. Despite these complications, aligning theclassical/instrumental distinction with the prediction/control distinction is a convenient
first approximation in connecting reinforcement learning to animal learning.
In psychology, the term reinforcement is used to describe learning in both classical andinstrumental conditioning. Originally referring only to the strengthening of a pattern of
behavior, it is frequently also used for the weakening of a pattern of behavior. A stimulus
considered to be the cause of the change in behavior is called a reinforcer, whether or
not it is contingent on the animal’s previous behavior. At the end of this chapter we
discuss this terminology in more detail and how it relates to terminology used in machinelearning.

14.2

Classical Conditioning

While studying the activity of the digestive system, the celebrated Russian physiologist
Ivan Pavlov found that an animal’s innate responses to certain triggering stimuli cancome to be triggered by other stimuli that are quite unrelated to the inborn triggers. His
experimental subjects were dogs that had undergone minor surgery to allow the intensityof their salivary reflex to be accurately measured. In one case he describes, the dog did
not salivate under most circumstances, but about 5 seconds after being presented withfood it produced about six drops of saliva over the next several seconds. After several
repetitions of presenting another stimulus, one not related to food, in this case the soundof a metronome, shortly before the introduction of food, the dog salivated in response to
the sound of the metronome in the same way it did to the food. “The activity of thesalivary gland has thus been called into play by impulses of sound—a stimulus quitealien to food” (Pavlov, 1927, p. 22). Summarizing the significance of this finding, Pavlov
wrote:
It is pretty evident that under natural conditions the normal animal must
respond not only to stimuli which themselves bring immediate benefit orharm, but also to other physical or chemical agencies—waves of sound, light,
and the like—which in themselves only signal the approach of these stimuli;
though it is not the sight and sound of the beast of prey which is in itself
harmful to the smaller animal, but its teeth and claws. (Pavlov, 1927, p. 14)
Connecting new stimuli to innate reflexes in this way is now called classical, or Pavlovian,conditioning. Pavlov (or more exactly, his translators) called inborn responses (e.g.,344

Chapter 14: Psychology

salivation in his demonstration described above) “unconditioned responses” (URs), theirnatural triggering stimuli (e.g., food) “unconditioned stimuli” (USs), and new responses
triggered by predictive stimuli (e.g., here also salivation) “conditioned responses” (CRs).
A stimulus that is initially neutral, meaning that it does not normally elicit strong
responses (e.g., the metronome sound), becomes a “conditioned stimulus” (CS) as theanimal learns that it predicts the US and so comes to produce a CR in response to the CS.
These terms are still used in describing classical conditioning experiments (though bettertranslations would have been “conditional” and “unconditional” instead of conditioned
and unconditioned). The US is called a reinforcer because it reinforces producing a CR
in response to the CS.
The arrangement of stimuli in two Delay Conditioning
common types of classical conditionCS
ing experiments is shown to the right.
In delay conditioning, the CS extends
USthroughout the interstimulus interval, or
ISI, which is the time interval between
the CS onset and the US onset (withISI
the CS ending when the US ends in acommon version shown here). In trace
Trace Conditioning
conditioning, the US begins after the CSCS
ends, and the time interval between CSo↵set and US onset is called the traceUSinterval.
The salivation of Pavlov’s dogs to thetsound of a metronome is just one example of classical conditioning, which has
been intensively studied across many response systems of many species of animals. URs
are often preparatory in some way, like the salivation of Pavlov’s dog, or protective insome way, like an eye blink in response to something irritating to the eye, or freezing
in response to seeing a predator. Experiencing the CS-US predictive relationship overa series of trials causes the animal to learn that the CS predicts the US so that the
animal can respond to the CS with a CR that prepares the animal for, or protects it
from, the predicted US. Some CRs are similar to the UR but begin earlier and di↵er inways that increase their e↵ectiveness. In one intensively studied type of experiment, forexample, a tone CS reliably predicts a pu↵ of air (the US) to a rabbit’s eye, triggering a
UR consisting of the closure of a protective inner eyelid called the nictitating membrane.
After one or more trials, the tone comes to trigger a CR consisting of membrane closurethat begins before the air pu↵ and eventually becomes timed so that peak closure occurs
just when the air pu↵ is likely to occur. This CR, being initiated in anticipation of the
air pu↵ and appropriately timed, o↵ers better protection than simply initiating closure
as a reaction to the irritating US. The ability to act in anticipation of important eventsby learning about predictive relationships among stimuli is so beneficial that it is widely
present across the animal kingdom.

14.2. Classical Conditioning

14.2.1345

Blocking and Higher-order Conditioning

Many interesting properties of classical conditioning have been observed in experiments.
Beyond the anticipatory nature of CRs, two widely observed properties figured prominently
in the development of classical conditioning models: blocking and higher-order conditioning.
Blocking occurs when an animal fails to learn a CR when a potential CS is presented alongwith another CS that had been used previously to condition the animal to produce that
CR. For example, in the first stage of a blocking experiment involving rabbit nictitating
membrane conditioning, a rabbit is first conditioned with a tone CS and an air pu↵ US
to produce the CR of closing its nictitating membrane in anticipation of the air pu↵. The
experiment’s second stage consists of additional trials in which a second stimulus, say
a light, is added to the tone to form a compound tone/light CS followed by the sameair pu↵ US. In the experiment’s third phase, the second stimulus alone—the light—ispresented to the rabbit to see if the rabbit has learned to respond to it with a CR. It
turns out that the rabbit produces very few, or no, CRs in response to the light: learning
to the light had been blocked by the previous learning to the tone.2 Blocking results likethis challenged the idea that conditioning depends only on simple temporal contiguity,that is, that a necessary and sufficient condition for conditioning is that a US frequently
follows a CS closely in time. In the next section we describe the Rescorla–Wagner model(Rescorla and Wagner, 1972) that o↵ered an influential explanation for blocking.
Higher-order conditioning occurs when a previously-conditioned CS acts as a US
in conditioning another initially neutral stimulus. Pavlov described an experiment inwhich his assistant first conditioned a dog to salivate to the sound of a metronome thatpredicted a food US, as described above. After this stage of conditioning, a number oftrials were conducted in which a black square, to which the dog was initially indi↵erent,
was placed in the dog’s line of vision followed by the sound of the metronome—and
this was not followed by food. In just ten trials, the dog began to salivate merely uponseeing the black square, despite the fact that the sight of it had never been followed byfood. The sound of the metronome itself acted as a US in conditioning a salivation CR
to the black square CS. This was second-order conditioning. If the black square had
been used as a US to establish salivation CRs to another otherwise neutral CS, it would
have been third-order conditioning, and so on. Higher-order conditioning is difficult todemonstrate, especially above the second order, in part because a higher-order reinforcer
loses its reinforcing value due to not being repeatedly followed by the original US during
higher-order conditioning trials. But under the right conditions, such as intermixing
first-order trials with higher-order trials or by providing a general energizing stimulus,
higher-order conditioning beyond the second order can be demonstrated. As we describe
below, the TD model of classical conditioning uses the bootstrapping idea that is central
to our approach to extend the Rescorla–Wagner model’s account of blocking to include
both the anticipatory nature of CRs and higher-order conditioning.

2 Comparison with a control group is necessary to show that the previous conditioning to the tone is
responsible for blocking learning to the light. This is done by trials with the tone/light CS but with noprior conditioning to the tone. Learning to the light in this case is unimpaired. Moore and Schmajuk(2008) give a full account of this procedure.

346

Chapter 14: Psychology

Higher-order instrumental conditioning occurs as well. In this case, a stimulus thatconsistently predicts primary reinforcement becomes a reinforcer itself, where reinforcement is primary if its rewarding or penalizing quality has been built into the animal byevolution. The predicting stimulus becomes a secondary reinforcer, or more generally, ahigher-order or conditioned reinforcer —the latter being a better term when the predicted
reinforcing stimulus is itself a secondary, or an even higher-order, reinforcer. A conditioned reinforcer delivers conditioned reinforcement: conditioned reward or conditioned
penalty. Conditioned reinforcement acts like primary reinforcement in increasing an
animal’s tendency to produce behavior that leads to conditioned reward, and to decrease
an animal’s tendency to produce behavior that leads to conditioned penalty. (See ourcomments at the end of this chapter that explain how our terminology sometimes di↵ers,
as it does here, from terminology used in psychology.)
Conditioned reinforcement is a key phenomenon that explains, for instance, why wework for the conditioned reinforcer money, whose worth derives solely from what ispredicted by having it. In actor–critic methods described in Section 13.5 (and discussed
in the context of neuroscience in Sections 15.7 and 15.8), the critic uses a TD method
to evaluate the actor’s policy, and its value estimates provide conditioned reinforcement
to the actor, allowing the actor to improve its policy. This analog of higher-orderinstrumental conditioning helps address the credit-assignment problem mentioned in
Section 1.7 because the critic gives moment-by-moment reinforcement to the actor when
the primary reward signal is delayed. We discuss this more below in Section 14.4.

14.2.2The Rescorla–Wagner Model

Rescorla and Wagner created their model mainly to account for blocking. The coreidea of the Rescorla–Wagner model is that an animal only learns when events violate
its expectations, in other words, only when the animal is surprised (although without
necessarily implying any conscious expectation or emotion). We first present Rescorla andWagner’s model using their terminology and notation before shifting to the terminology
and notation we use to describe the TD model.
Here is how Rescorla and Wagner described their model. The model adjusts the“associative strength” of each component stimulus of a compound CS, which is a number
representing how strongly or reliably that component is predictive of a US. Whena compound CS consisting of several component stimuli is presented in a classicalconditioning trial, the associative strength of each component stimulus changes in a waythat depends on an associative strength associated with the entire stimulus compound,called the “aggregate associative strength,” and not just on the associative strength ofeach component itself.
Rescorla and Wagner considered a compound CS AX, consisting of component stimuli
A and X, where the animal may have already experienced stimulus A, and stimulus X
might be new to the animal. Let VA , VX , and VAX respectively denote the associative
strengths of stimuli A, X, and the compound AX. Suppose that on a trial the compound
CS AX is followed by a US, which we label stimulus Y. Then the associative strengths of14.2. Classical Conditioning347

the stimulus components change according to these expressions:
VA = ↵A Y (RYVAX )VX = ↵X Y (RY

VAX ),

where ↵A Y and ↵X Y are the step-size parameters, which depend on the identities of
the CS components and the US, and RY is the asymptotic level of associative strengththat the US Y can support. (Rescorla and Wagner used here instead of R, but we
use R to avoid confusion with our use of and because we usually think of this as themagnitude of a reward signal, with the caveat that the US in classical conditioning is notnecessarily rewarding or penalizing.) A key assumption of the model is that the aggregate
associative strength VAX is equal to VA + VX . The associative strengths as changed bythese s become the associative strengths at the beginning of the next trial.
To be complete, the model needs a response-generation mechanism, which is a wayof mapping values of V s to CRs. Because this mapping would depend on details of
the experimental situation, Rescorla and Wagner did not specify a mapping but simply
assumed that larger V s would produce stronger or more likely CRs, and that negative
V s would mean that there would be no CRs.
The Rescorla–Wagner model accounts for the acquisition of CRs in a way that explains
blocking. As long as the aggregate associative strength, VAX , of the stimulus compoundis below the asymptotic level of associative strength, RY , that the US Y can support, theprediction error RY VAX is positive. This means that over successive trials the associative
strengths VA and VX of the component stimuli increase until the aggregate associative
strength VAX equals RY , at which point the associative strengths stop changing (unless the
US changes). When a new component is added to a compound CS to which the animal has
already been conditioned, further conditioning with the augmented compound produces
little or no increase in the associative strength of the added CS component because theerror has already been reduced to zero, or to a low value. The occurrence of the US isalready predicted nearly perfectly, so little or no error—or surprise—is introduced by the
new CS component. Prior learning blocks learning to the new component.
To transition from Rescorla and Wagner’s model to the TD model of classical conditioning (which we just call the TD model), we first recast their model in terms of theconcepts that we are using throughout this book. Specifically, we match the notation
we use for learning with linear function approximation (Section 9.4), and we think of
the conditioning process as one of learning to predict the “magnitude of the US” on a
trial on the basis of the compound CS presented on that trial, where the magnitude ofa US Y is the RY of the Rescorla–Wagner model as given above. We also introduce
states. Because the Rescorla–Wagner model is a trial-level model, meaning that it deals
with how associative strengths change from trial to trial without considering any details
about what happens within and between trials, we do not have to consider how stateschange during a trial until we present the full TD model in the following section. Instead,here we simply think of a state as a way of labeling a trial in terms of the collection ofcomponent CSs that are present on the trial.
Therefore, assume that trial-type, or state, s is described by a real-valued vector offeatures x(s) = (x1 (s), x2 (s), . . . , xd (s))> where xi (s) = 1 if CSi , the ith component of a

348

Chapter 14: Psychology

compound CS, is present on the trial and 0 otherwise. Then if the d-dimensional vectorof associative strengths is w, the aggregate associative strength for trial-type s isv̂(s,w) = w> x(s).

(14.1)

This corresponds to a value estimate in reinforcement learning, and we think of it as the
US prediction.
Now temporally let t denote the number of a complete trial and not its usual meaning
as a time step (we revert to t’s usual meaning when we extend this to the TD model
below), and assume that St is the state corresponding to trial t. Conditioning trial t
updates the associative strength vector wt to wt+1 as follows:
wt+1 = wt + ↵ t x(St ),(14.2)

where ↵ is the step-size parameter, and—because here we are describing the Rescorla–
Wagner model— t is the prediction error
t = Rtv̂(St ,wt ).

(14.3)

Rt is the target of the prediction on trial t, that is, the magnitude of the US, or in Rescorla
and Wagner’s terms, the associative strength that the US on the trial can support. Notethat because of the factor x(St ) in (14.2), only the associative strengths of CS components
present on a trial are adjusted as a result of that trial. You can think of the prediction
error as a measure of surprise, and the aggregate associative strength as the animal’s
expectation that is violated when it does not match the target US magnitude.
From the perspective of machine learning, the Rescorla–Wagner model is an errorcorrection supervised learning rule. It is essentially the same as the Least Mean Square(LMS), or Widrow-Ho↵, learning rule (Widrow and Ho↵, 1960) that finds the weights—
here the associative strengths—that make the average of the squares of all the errors as
close to zero as possible. It is a “curve-fitting,” or regression, algorithm that is widelyused in engineering and scientific applications (see Section 9.4).3
The Rescorla–Wagner model was very influential in the history of animal learningtheory because it showed that a “mechanistic” theory could account for the main facts
about blocking without resorting to more complex cognitive theories involving, forexample, an animal’s explicit recognition that another stimulus component had been
added and then scanning its short-term memory backward to reassess the predictive
relationships involving the US. The Rescorla–Wagner model showed how traditional
contiguity theories of conditioning—that temporal contiguity of stimuli was a necessary
and sufficient condition for learning—could be adjusted in a simple way to account for
blocking (Moore and Schmajuk, 2008).
The Rescorla–Wagner model provides a simple account of blocking and some otherfeatures of classical conditioning, but it is not a complete or perfect model of classical
3 The only di↵erences between the LMS rule and the Rescorla–Wagner model are that for LMS the
input vectors xt can have any real numbers as components, and—at least in the simplest version of the
LMS rule—the step-size parameter ↵ does not depend on the input vector or the identity of the stimulus
setting the prediction target.

14.2. Classical Conditioning349

conditioning. Di↵erent ideas account for a variety of other observed e↵ects, and progressis still being made toward understanding the many subtleties of classical conditioning.
The TD model, which we describe next, though also not a complete or perfect model
model of classical conditioning, extends the Rescorla–Wagner model to address how
within-trial and between-trial timing relationships among stimuli can influence learningand how higher-order conditioning might arise.

14.2.3

The TD Model

The TD model is a real-time model, as opposed to a trial-level model like the Rescorla–
Wagner model. A single step t in the Rescorla–Wagner model represents an entire
conditioning trial. The model does not apply to details about what happens during thetime a trial is taking place, or what might happen between trials. Within each trial an
animal might experience various stimuli whose onsets occur at particular times and thathave particular durations. These timing relationships strongly influence learning. The
Rescorla–Wagner model also does not include a mechanism for higher-order conditioning,whereas for the TD model, higher-order conditioning is a natural consequence of thebootstrapping idea that is at the base of TD algorithms.
To describe the TD model we begin with the formulation of the Rescorla–Wagnermodel above, but t now labels time steps within or between trials instead of complete
trials. Think of the time between t and t + 1 as a small time interval, say .01 second, andthink of a trial as a sequences of states, one associated with each time step, where thestate at step t now represents details of how stimuli are represented at t instead of justa label for the CS components present on a trial. In fact, we can completely abandon
the idea of trials. From the point of view of the animal, a trial is just a fragment of its
continuing experience interacting with its world. Following our usual view of an agent
interacting with its environment, imagine that the animal is experiencing an endlesssequence of states s, each represented by a feature vector x(s). That said, it is still often
convenient to refer to trials as fragments of time during which patterns of stimuli repeat
in an experiment.
State features are not restricted to describing the external stimuli that an animal
experiences; they can describe neural activity patterns that external stimuli produce inan animal’s brain, and these patterns can be history-dependent, meaning that they canbe persistent patterns produced by sequences of external stimuli. Of course, we do notknow exactly what these neural activity patterns are, but a real-time model like the TD
model allows one to explore the consequences on learning of di↵erent hypotheses about
the internal representations of external stimuli. For these reasons, the TD model does
not commit to any particular state representation. In addition, because the TD model
includes discounting and eligibility traces that span time intervals between stimuli, themodel also makes it possible to explore how discounting and eligibility traces interact withstimulus representations in making predictions about the results of classical conditioning
experiments.
Below we describe some of the state representations that have been used with the
TD model and some of their implications, but for the moment we stay agnostic about

350

Chapter 14: Psychology

the representation and just assume that each state s is represented by a feature vector
x(s) = (x1 (s), x2 (s), . . . , xn (s))> . Then the aggregate associative strength corresponding
to a state s is given by (14.1), the same as for the Rescorla-Wagner model, but the TD
model updates the associative strength vector, w, di↵erently. With t now labeling a time
step instead of a complete trial, the TD model governs learning according to this update:
wt+1 = wt + ↵ t zt ,(14.4)

which replaces xt (St ) in the Rescorla–Wagner update (14.2) with zt , a vector of eligibility
traces, and instead of the t of (14.3), here t is a TD error:t = Rt+1 +v̂(St+1 ,wt )v̂(St ,wt ),(14.5)

where is a discount factor (between 0 and 1), Rt is the prediction target at time t, andv̂(St+1 ,wt ) and v̂(St ,wt ) are aggregate associative strengths at t + 1 and t as defined by(14.1).
Each component i of the eligibility-trace vector zt increments or decrements according
to the component xi (St ) of the feature vector x(St ), and otherwise decays with a rate
determined by :zt =zt 1 + x(St ).

(14.6)

Here is the usual eligibility trace decay parameter.
Note that if = 0, the TD model reduces to the Rescorla–Wagner model with the
exceptions that: the meaning of t is di↵erent in each case (a trial number for the
Rescorla–Wagner model and a time step for the TD model), and in the TD model there
is a one-time-step lead in the prediction target R. The TD model is equivalent to thebackward view of the semi-gradient TD( ) algorithm with linear function approximation(Chapter 12), except that Rt in the model does not have to be a reward signal as it does
when the TD algorithm is used to learn a value function for policy-improvement.

14.2.4

TD Model Simulations

Real-time conditioning models like the TD model are interesting primarily because they
make predictions for a wide range of situations that cannot be represented by trial-level
models. These situations involve the timing and durations of conditionable stimuli, thetiming of these stimuli in relation to the timing of the US, and the timing and shapesof CRs. For example, the US generally must begin after the onset of a neutral stimulus
for conditioning to occur, with the rate and e↵ectiveness of learning depending on theinter-stimulus interval, or ISI, the interval between the onsets of the CS and the US. WhenCRs appear, they generally begin before the appearance of the US and their temporalprofiles change during learning. In conditioning with compound CSs, the component
stimuli of the compound CSs may not all begin and end at the same time, sometimes
forming what is called a serial compound in which the component stimuli occur in asequence over time. Timing considerations like these make it important to consider how14.2. Classical Conditioning
Complete Serial
Microstimuli
Compound

351

Presence

CS

Stimulus
Representation

US

Figure 14.1: Three stimulus representations (in columns) sometimes used with the TD model.
Each row represents one element of the stimulus representation. The three representations varyalong a temporal generalization gradient, with no generalization between nearby time points inthe complete serial compound (left column) and complete generalization between nearby time
points in the presence representation (right column). The microstimulus representation occupies
a middle ground. The degree of temporal generalization determines the temporal granularity
with which US predictions are learned. Adapted with minor changes from Learning & Behavior,
Evaluating the TD Model of Classical Conditioning, volume 40, 2012, p. 311, E. A. Ludvig, R. S.
Sutton, E. J. Kehoe. With permission of Springer.

stimuli are represented, how these representations unfold over time during and between
trials, and how they interact with discounting and eligibility traces.
Figure 14.1 shows three of the stimulus representations that have been used in exploring
the behavior of the TD model: the complete serial compound (CSC), the microstimulus(MS), and the presence representations (Ludvig, Sutton, and Kehoe, 2012). These
representations di↵er in the degree to which they force generalization among nearby time
points during which a stimulus is present.
The simplest of the representations shown in Figure 14.1 is the presence representationin the figure’s right column. This representation has a single feature for each component
CS present on a trial, where the feature has value 1 whenever that component is present,
and 0 otherwise.4 The presence representation is not a realistic hypothesis about how
stimuli are represented in an animal’s brain, but as we describe below, the TD modelwith this representation can produce many of the timing phenomena seen in classicalconditioning.
4 In our formalism, there is a di↵erent state, S , for each time step t during a trial, and for a trialtin which a compound CS consists of n component CSs of various durations occurring at various times
throughout the trial, there is a feature, xi , for each component CSi , i = 1, . . . , n, where xi (St ) = 1 for
all times t when the CSi is present, and equals zero otherwise.

352

Chapter 14: Psychology

For the CSC representation (left column of Figure 14.1), the onset of each external
stimulus initiates a sequence of precisely-timed short-duration internal signals that
continues until the external stimulus ends.5 This is like assuming the animal’s nervous
system has a clock that keeps precise track of time during stimulus presentations; it iswhat engineers call a “tapped delay line.” Like the presence representation, the CSC
representation is unrealistic as a hypothesis about how the brain internally represents
stimuli, but Ludvig et al. (2012) call it a “useful fiction” because it can reveal details of
how the TD model works when relatively unconstrained by the stimulus representation.
The CSC representation is also used in most TD models of dopamine-producing neurons
in the brain, a topic we take up in Chapter 15. The CSC representation is often viewed
as an essential part of the TD model, although this view is mistaken.
The MS representation (center column of Figure 14.1) is like the CSC representationin that each external stimulus initiates a cascade of internal stimuli, but in this case theinternal stimuli—the microstimuli—are not of such limited and non-overlapping form;
they are extended over time and overlap. As time elapses from stimulus onset, di↵erent
sets of microstimuli become more or less active, and each subsequent microstimulus
becomes progressively wider in time and reaches a lower maximal level. Of course, there
are many MS representations depending on the nature of the microstimuli, and a numberof examples of MS representations have been studied in the literature, in some cases alongwith proposals for how an animal’s brain might generate them (see the Bibliographic andHistorical Comments at the end of this chapter). MS representations are more realistic
than the presence or CSC representations as hypotheses about neural representations ofstimuli, and they allow the behavior of the TD model to be related to a broader collectionof phenomena observed in animal experiments. In particular, by assuming that cascadesof microstimuli are initiated by USs as well as by CSs, and by studying the significant
e↵ects on learning of interactions between microstimuli, eligibility traces, and discounting,
the TD model is helping to frame hypotheses to account for many of the subtle phenomenaof classical conditioning and how an animal’s brain might produce them. We say more
about this below, particularly in Chapter 15 where we discuss reinforcement learning andneuroscience.
Even with the simple presence representation, however, the TD model produces all thebasic properties of classical conditioning that are accounted for by the Rescorla–Wagnermodel, plus features of conditioning that are beyond the scope of trial-level models. For
example, as we have already mentioned, a conspicuous feature of classical conditioning isthat the US generally must begin after the onset of a neutral stimulus for conditioning
to occur, and that after conditioning, the CR begins before the appearance of the US.
In other words, conditioning generally requires a positive ISI, and the CR generally
anticipates the US. How the strength of conditioning (e.g., the percentage of CRs elicitedby a CS) depends on the ISI varies substantially across species and response systems, but
it typically has the following properties: it is negligible for a zero or negative ISI, i.e., when
5 In our formalism, for each CS component CS present on a trial, and for each time step t during aitrial, there is a separate feature xti , where xti (St0 ) = 1 if t = t0 for any t0 at which CSi is present, andequals 0 otherwise. This is di↵erent from the CSC representation in Sutton and Barto (1990) in which
there are the same distinct features for each time step but no reference to external stimuli; hence thename complete serial compound.

14.2. Classical Conditioning

353

the US onset occurs simultaneously with, or earlier than, the CS onset (although research
has found that associative strengths sometimes increase slightly or become negative withnegative ISIs); it increases to a maximum at a positive ISI where conditioning is moste↵ective; and it then decreases to zero after an interval that varies widely with response
systems. The precise shape of this dependency for the TD model depends on the valuesof its parameters and details of the stimulus representation, but these basic features of
ISI-dependency are core properties of the TD model.
One of the theoretical issues arisingwith serial-compound conditioning, thatis, conditioning with a compound CS
whose components occur in a sequence,
concerns the facilitation of remote associations. It has been found that if theempty trace interval between a first CS(CSA) and the US is filled with a second
CS (CSB) to form a serial-compound
stimulus, then conditioning to CSA is
facilitated. Shown to the right is the
behavior of the TD model with the presence representation in a simulation ofsuch an experiment whose timing details
are shown above. Consistent with theexperimental results (Kehoe, 1982), themodel shows facilitation of both the rate Facilitation of remote associations in the TD modelof conditioning and the asymptotic levelof conditioning of the first CS due to thepresence of the second CS.
A well-known demonstration of thee↵ects on conditioning of temporal relationships among stimuli within a trial
is an experiment by Egger and Miller(1962) that involved two overlapping
CSs in a delay configuration as shown
to the right (top). Although CSB wasin a better temporal relationship with
the US, the presence of CSA substantially reduced conditioning to CSB ascompared to controls in which CSA was wCSB
absent. Directly to the right is shown
the same result being generated by the
TD model in a simulation of this experiment with the presence representation.
The TD model accounts for blocking
The Egger-Miller e↵ect in the TD model
because it is an error-correcting learning

354

Chapter 14: Psychology

rule like the Rescorla–Wagner model. Beyond accounting for basic blocking results,
however, the TD model predicts (with the presence representation and more complex
representations as well) that blocking is reversed if the blocked stimulus is moved earlier
in time (like CSA in the diagram to theright) so that its onset occurs before theonset of the blocking stimulus. This featureof the TD model’s behavior deserves attention because it had not been observed at
the time of the model’s introduction. Recall
that in blocking, if an animal has alreadylearned that one CS predicts a US, thenlearning that a newly-added second CS also
predicts the US is much reduced, i.e., isblocked. But if the newly-added second CS wCSB
begins earlier than the pretrained CS, then—according to the TD model—learning to the
newly-added CS is not blocked. In fact, as
training continues and the newly-added CS
gains associative strength, the pretrained
CS loses associative strength. The behavior Figure 14.2: Temporal primacy overridingof the TD model under these conditions blocking in the TD model.
is shown in the lower part of Figure 14.2.
This simulation experiment di↵ered from the Egger-Miller experiment (bottom of thepreceding page) in that the shorter CS with the later onset was given prior training
until it was fully associated with the US. This surprising prediction led Kehoe, Schreurs,
and Graham (1987) to conduct the experiment using the well-studied rabbit nictitating
membrane preparation. Their results confirmed the model’s prediction, and they notedthat non-TD models have considerable difficulty explaining their data.
With the TD model, an earlier predictive stimulus takes precedence over a later
predictive stimulus because, like all the prediction methods described in this book, theTD model is based on the backing-up or bootstrapping idea: updates to associative
strengths shift the strengths at a particular state toward the strength at later states.
Another consequence of bootstrapping is that the TD model provides an account of higherorder conditioning, a feature of classical conditioning that is beyond the scope of the
Rescorla-Wagner and similar models. As we described above, higher-order conditioningis the phenomenon in which a previously-conditioned CS can act as a US in conditioning
another initially neutral stimulus. Figure 14.3 shows the behavior of the TD model (again
with the presence representation) in a higher-order conditioning experiment—in this case
it is second-order conditioning. In the first phase (not shown in the figure), CSB is trained
to predict a US so that its associative strength increases, here to 1.65. In the second
phase, CSA is paired with CSB in the absence of the US, in the sequential arrangement
shown at the top of the figure. CSA acquires associative strength even though it is neverpaired with the US. With continued training, CSA’s associative strength reaches a peak
and then decreases because the associative strength of CSB, the secondary reinforcer,
decreases so that it loses its ability to provide secondary reinforcement. CSB’s associative

14.2. Classical Conditioning

355

strength decreases because the US does notoccur in these higher-order conditioning trials. These are extinction trials for CSB
because its predictive relationship to the
US is disrupted so that its ability to act asa reinforcer decreases. This same patternis seen in animal experiments. This extinction of conditioned reinforcement in higherorder conditioning trials makes it difficult
wwto demonstrate higher-order conditioning
unless the original predictive relationships
are periodically refreshed by occasionallyinserting first-order trials.
The TD model produces an analog of
second- and higher-order conditioning beFigure 14.3: Second-order conditioning with cause v̂(St+1 ,wt )v̂(St ,wt ) appears in
the TD model.
the TD error t (14.5). Due to the firstphase of learning, v̂(St+1 ,wt ) may di↵er
from v̂(St ,wt ), making t non-zero (a temporal di↵erence). This di↵erence has the same
status as Rt+1 in (14.5), implying that as far as learning is concerned there is no di↵erence
between a temporal di↵erence and the occurrence of a US. In fact, this feature of the
TD algorithm is one of the major reasons for its development, which we now understandthrough its connection to dynamic programming as described in Chapter 6. Bootstrapping
values is intimately related to second-order, and higher-order, conditioning.
In the examples of the TD model’s behavior described above, we examined only thechanges in the associative strengths of the CS components; we did not look at what
the model predicts about properties of an animal’s conditioned responses (CRs): theirtiming, shape, and how they develop over conditioning trials. These properties dependon the species, the response system being observed, and parameters of the conditioning
trials, but in many experiments with di↵erent animals and di↵erent response systems, themagnitude of the CR, or the probability of a CR, increases as the expected time of the
US approaches. For example, in classical conditioning of a rabbit’s nictitating membrane
response that we mentioned above, over conditioning trials the delay from CS onset towhen the nictitating membrane begins to move across the eye decreases over trials, andthe amplitude of this anticipatory closure gradually increases over the interval between
the CS and the US until the membrane reaches maximal closure at the expected time of
the US. The timing and shape of this CR is critical to its adaptive significance—covering
the eye too early reduces vision (even though the nictitating membrane is translucent),
while covering it too late is of little protective value. Capturing CR features like these ischallenging for models of classical conditioning.
The TD model does not include as part of its definition any mechanism for translating the time course of the US prediction, v̂(St ,wt ), into a profile that can be compared356

Chapter 14: Psychology

v̂

v̂

v̂

with the properties of an animal’s CR. The simplest choice is to let the time course ofa simulated CR equal the time course of the US prediction. In this case, features ofsimulated CRs and how they change over trials depend only on the stimulus representation
chosen and the values of the model’s parameters ↵, , and .
Figure 14.4 shows the time courses of US predictions at di↵erent points during learningwith the three representations shown in Figure 14.1. For these simulations the US
occurred 25 time steps after the onset of the CS, and ↵ = .05, = .95 and = .97.
With the CSC representation (Figure 14.4 left), the curve of the US prediction formedby the TD model increases exponentially throughout the interval between the CS andthe US until it reaches a maximum exactly when the US occurs (at time step 25). This
exponential increase is the result of discounting in the TD model learning rule. With thepresence representation (Figure 14.4 middle), the US prediction is nearly constant while
the stimulus is present because there is only one weight, or associative strength, to belearned for each stimulus. Consequently, the TD model with the presence representation
cannot recreate many features of CR timing. With an MS representation (Figure 14.4
right), the development of the TD model’s US prediction is more complicated. After 200
trials the prediction’s profile is a reasonable approximation of the US prediction curve
produced with the CSC representation.

Figure 14.4: Time course of US prediction over the course of acquisition for the TD modelwith three di↵erent stimulus representations. Left: With the complete serial compound (CSC),
the US prediction increases exponentially through the interval, peaking at the time of the US.
At asymptote (trial 200), the US prediction peaks at the US intensity (1 in these simulations).
Middle: With the presence representation, the US prediction converges to an almost constantlevel. This constant level is determined by the US intensity and the length of the CS–US interval.
Right: With the microstimulus representation, at asymptote, the TD model approximates the
exponentially increasing time course depicted with the CSC through a linear combination of thedi↵erent microstimuli. Adapted with minor changes from Learning & Behavior, Evaluating the
TD Model of Classical Conditioning, volume 40, 2012, E. A. Ludvig, R. S. Sutton, E. J. Kehoe.
With permission of Springer.

The US prediction curves shown in Figure 14.4 were not intended to precisely match
profiles of CRs as they develop during conditioning in any particular animal experiment,
but they illustrate the strong influence that the stimulus representation has on predictions
derived from the TD model. Further, although we can only mention it here, how the14.3. Instrumental Conditioning

357

stimulus representation interacts with discounting and eligibility traces is importantin determining properties of the US prediction profiles produced by the TD model.
Another dimension beyond what we can discuss here is the influence of di↵erent responsegeneration mechanisms that translate US predictions into CR profiles; the profiles shownin Figure 14.4 are “raw” US prediction profiles. Even without any special assumption
about how an animal’s brain might produce overt responses from US predictions, however,the profiles in Figure 14.4 for the CSC and MS representations increase as the time of the
US approaches and reach a maximum at the time of the US, as is seen in many animal
conditioning experiments.
The TD model, when combined with particular stimulus representations and responsegeneration mechanisms, is able to account for a surprisingly wide range of phenomena
observed in animal classical conditioning experiments, but it is far from being a perfectmodel. To generate other details of classical conditioning the model needs to be extended,
perhaps by adding model-based elements and mechanisms for adaptively altering some ofits parameters. Other approaches to modeling classical conditioning depart significantly
from the Rescorla–Wagner-style error-correction process. Bayesian models, for example,
work within a probabilistic framework in which experience revises probability estimates.
All of these models usefully contribute to our understanding of classical conditioning.
Perhaps the most notable feature of the TD model is that it is based on a theory—thetheory we have described in this book—that suggests an account of what an animal’s
nervous system is trying to do while undergoing conditioning: it is trying to form accurate
long-term predictions, consistent with the limitations imposed by the way stimuli are
represented and how the nervous system works. In other words, it suggests a normative
account of classical conditioning in which long-term, instead of immediate, prediction is a
key feature.
The development of the TD model of classical conditioning is one instance in which theexplicit goal was to model some of the details of animal learning behavior. In addition toits standing as an algorithm, then, TD learning is also the basis of this model of aspectsof biological learning. As we discuss in Chapter 15, TD learning has also turned out
to underlie an influential model of the activity of neurons that produce dopamine, achemical in the brain of mammals that is deeply involved in reward processing. These
are instances in which reinforcement learning theory makes detailed contact with animal
behavioral and neural data.
We now turn to considering correspondences between reinforcement learning and animal
behavior in instrumental conditioning experiments, the other major type of laboratory
experiment studied by animal learning psychologists.

14.3

Instrumental Conditioning

In instrumental conditioning experiments learning depends on the consequences of behavior: the delivery of a reinforcing stimulus is contingent on what the animal does.
In classical conditioning experiments, in contrast, the reinforcing stimulus—the US—isdelivered independently of the animal’s behavior. Instrumental conditioning is usuallyconsidered to be the same as operant conditioning, the term B. F. Skinner (1938, 1963)

358

Chapter 14: Psychology

introduced for experiments with behavior-contingent reinforcement, though the experiments and theories of those who use these two terms di↵er in a number of ways, some ofwhich we touch on below. We will exclusively use the term instrumental conditioning forexperiments in which reinforcement is contingent upon behavior. The roots of instrumental conditioning go back to experiments performed by the American psychologist Edward
Thorndike one hundred years before publication of the first edition of this book.
Thorndike observed the behavior of catswhen they were placed in “puzzle boxes,”
such as the one at the right, from which
they could escape by appropriate actions.
For example, a cat could open the doorof one box by performing a sequence of
three separate actions: depressing a platform at the back of the box, pulling a stringby clawing at it, and pushing a bar up or
down. When first placed in a puzzle box,with food visible outside, all but a few ofThorndike’s cats displayed “evident signs
One of Thorndike’s puzzle boxes.
of discomfort” and extraordinarily vigorous Reprinted from Thorndike, Animal Intelligence: An
activity “to strive instinctively to escape Experimental Study of the Associative Processes inAnimals, The Psychological Review, Series of Monofrom confinement” (Thorndike, 1898).
graph Supplements II(4), Macmillan, New York, 1898.
In experiments with di↵erent cats andboxes with di↵erent escape mechanisms, Thorndike recorded the amounts of time eachcat took to escape over multiple experiences in each box. He observed that the time
almost invariably decreased with successive experiences, for example, from 300 seconds
to 6 or 7 seconds. He described cats’ behavior in a puzzle box like this:The cat that is clawing all over the box in her impulsive struggle will probably
claw the string or loop or button so as to open the door. And gradually all theother non-successful impulses will be stamped out and the particular impulse
leading to the successful act will be stamped in by the resulting pleasure,
until, after many trials, the cat will, when put in the box, immediately claw
the button or loop in a definite way. (Thorndike 1898, p. 13)
These and other experiments (some with dogs, chicks, monkeys, and even fish) led
Thorndike to formulate a number of “laws” of learning, the most influential being theLaw of E↵ect. This law describes what is generally known as learning by trial anderror. As we mentioned in Chapter 1, many aspects of the Law of E↵ect have generated
controversy, and its details have been modified over the years. Still the law—in one formor another—expresses an enduring principle of learning.
Essential features of reinforcement learning algorithms correspond to features of animallearning described by the Law of E↵ect. First, reinforcement learning algorithms are
selectional, meaning that they try alternatives and select among them by comparing theirconsequences. Second, reinforcement learning algorithms are associative, meaning thatthe alternatives found by selection are associated with particular situations, or states,
to form the agent’s policy. Like learning described by the Law of E↵ect, reinforcement14.3. Instrumental Conditioning359

learning is not just the process of finding actions that produce a lot of reward, but alsoof connecting these actions to situations or states. Thorndike used the phrase learningby “selecting and connecting” (Hilgard, 1956). Natural selection in evolution is a primeexample of a selectional process, but it is not associative (at least as it is commonly
understood); supervised learning is associative, but it is not selectional because it relies
on instructions that directly tell the agent how to change its behavior.
In computational terms, the Law of E↵ect describes an elementary way of combining
search and memory: search in the form of trying and selecting among many actions
in each situation, and memory in the form of associations linking situations with theactions found—so far—to work best in those situations. Search and memory are essential
components of all reinforcement learning algorithms, whether memory takes the form ofan agent’s policy, value function, or environment model.
A reinforcement learning algorithm’s need to search means that it has to explore insome way. Animals clearly explore as well, and early animal learning researchers disagreed
about the degree of guidance an animal uses in selecting its actions in situations likeThorndike’s puzzle boxes. Are actions the result of “absolutely random, blind groping”(Woodworth, 1938, p. 777), or is there some degree of guidance, either from prior learning,
reasoning, or other means? Although some thinkers, including Thorndike, seem to havetaken the former position, others favored more deliberate exploration. Reinforcement
learning algorithms allow wide latitude for how much guidance an agent can employ inselecting actions. The forms of exploration we have used in the algorithms presented
in this book, such as "-greedy and upper-confidence-bound action selection, are merely
among the simplest. More sophisticated methods are possible, with the only stipulation
being that there has to be some form of exploration for the algorithms to work e↵ectively.
The feature of our treatment of reinforcement learning allowing the set of actions
available at any time to depend on the environment’s current state echoes somethingThorndike observed in his cats’ puzzle-box behaviors. The cats selected actions fromthose that they instinctively perform in their current situation, which Thorndike called
their “instinctual impulses.” First placed in a puzzle box, a cat instinctively scratches,claws, and bites with great energy: a cat’s instinctual responses to finding itself in aconfined space. Successful actions are selected from these and not from every possible
action or activity. This is like the feature of our formalism where the action selectedfrom a state s belongs to a set of admissible actions, A(s). Specifying these sets is an
important aspect of reinforcement learning because it can radically simplify learning.
They are like an animal’s instinctual impulses. On the other hand, Thorndike’s cats might
have been exploring according to an instinctual context-specific ordering over actions
rather than by just selecting from a set of instinctual impulses. This is another way tomake reinforcement learning easier.
Among the most prominent animal learning researchers influenced by the Law of E↵ectwere Clark Hull (e.g., Hull, 1943) and B. F. Skinner (e.g., Skinner, 1938). At the centerof their research was the idea of selecting behavior on the basis of its consequences.
Reinforcement learning has features in common with Hull’s theory, which included
eligibility-like mechanisms and secondary reinforcement to account for the ability to learn
when there is a significant time interval between an action and the consequent reinforcing

360

Chapter 14: Psychology

stimulus (see Section 14.4). Randomness also played a role in Hull’s theory through what
he called “behavioral oscillation” to introduce exploratory behavior.
Skinner did not fully subscribe to the memory aspect of the Law of E↵ect. Being averseto the idea of associative linkages, he instead emphasized selection from spontaneouslyemitted behavior. He introduced the term “operant” to emphasize the key role of anaction’s e↵ects on an animal’s environment. Unlike the experiments of Thorndike andothers, which consisted of sequences of separate trials, Skinner’s operant conditioning
experiments allowed animal subjects to behave for extended periods of time without
interruption. He invented the operant conditioning chamber, now called a “Skinner box,”
the most basic version of which contains a lever or key that an animal can press to obtaina reward, such as food or water, which would be delivered according to a well-defined rule,called a reinforcement schedule. By recording the cumulative number of lever presses
as a function of time, Skinner and his followers could investigate the e↵ect of di↵erent
reinforcement schedules on the animal’s rate of lever-pressing. Modeling results fromexperiments likes these using the reinforcement learning principles we present in thisbook is not well developed, but we mention some exceptions in the Bibliographic andHistorical Remarks section at the end of this chapter.
Another of Skinner’s contributions resulted from his recognition of the e↵ectiveness of
training an animal by reinforcing successive approximations of the desired behavior, aprocess he called shaping. Although this technique had been used by others, including
Skinner himself, its significance was impressed upon him when he and colleagues wereattempting to train a pigeon to bowl by swiping a wooden ball with its beak. After
waiting for a long time without seeing any swipe that they could reinforce, they... decided to reinforce any response that had the slightest resemblance toa swipe—perhaps, at first, merely the behavior of looking at the ball—and
then to select responses which more closely approximated the final form. The
result amazed us. In a few minutes, the ball was caroming o↵ the walls of
the box as if the pigeon had been a champion squash player. (Skinner, 1958,
p. 94)
Not only did the pigeon learn a behavior that is unusual for pigeons, it learned quickly
through an interactive process in which its behavior and the reinforcement contingencieschanged in response to each other. Skinner compared the process of altering reinforcement
contingencies to the work of a sculptor shaping clay into a desired form. Shaping is a
powerful technique for computational reinforcement learning systems as well. When it isdifficult for an agent to receive any non-zero reward signal at all, either due to sparsenessof rewarding situations or their inaccessibility given initial behavior, starting with aneasier problem and incrementally increasing its difficulty as the agent learns can be ane↵ective, and sometimes indispensable, strategy.
A concept from psychology that is especially relevant in the context of instrumental
conditioning is motivation, which refers to processes that influence the direction andstrength, or vigor, of behavior. Thorndike’s cats, for example, were motivated to escapefrom puzzle boxes because they wanted the food that was sitting just outside. Obtaining
this goal was rewarding to them and reinforced the actions allowing them to escape. It
is difficult to link the concept of motivation, which has many dimensions, in a precise

14.4. Delayed Reinforcement361

way to reinforcement learning’s computational perspective, but there are clear links withsome of its dimensions.
In one sense, a reinforcement learning agent’s reward signal is at the base of its
motivation: the agent is motivated to maximize the total reward it receives over the longrun. A key facet of motivation, then, is what makes an agent’s experience rewarding. In
reinforcement learning, reward signals depend on the state of the reinforcement learning
agent’s environment and the agent’s actions. Further, as pointed out in Chapter 1, thestate of the agent’s environment not only includes information about what is external tothe machine, like an organism or a robot, that houses the agent, but also what is internal
to this machine. Some internal state components correspond to what psychologists callan animal’s motivational state, which influences what is rewarding to the animal. For
example, an animal will be more rewarded by eating when it is hungry than when it has
just finished a satisfying meal. The concept of state dependence is broad enough to allow
for many types of modulating influences on the generation of reward signals.
Value functions provide a further link to psychologists’ concept of motivation. If themost basic motive for selecting an action is to obtain as much reward as possible, for areinforcement learning agent that selects actions using a value function, a more proximal
motive is to ascend the gradient of its value function, that is, to select actions expected
to lead to the most highly-valued next states (or what is essentially the same thing, toselect actions with the greatest action-values). For these agents, value functions are themain driving force determining the direction of their behavior.
Another dimension of motivation is that an animal’s motivational state not only
influences learning, but also influences the strength, or vigor, of the animal’s behavior
after learning. For example, after learning to find food in the goal box of a maze, a hungry
rat will run faster to the goal box than one that is not hungry. This aspect of motivation
does not link so cleanly to the reinforcement learning framework we present here, but
in the Bibliographical and Historical Remarks section at the end of this chapter we cite
several publications that propose theories of behavioral vigor based on reinforcement
learning.
We turn now to the subject of learning when reinforcing stimuli occur well after the
events they reinforce. The mechanisms used by reinforcement learning algorithms toenable learning with delayed reinforcement—eligibility traces and TD learning—closely
correspond to psychologists’ hypotheses about how animals can learn under these conditions.

14.4

Delayed Reinforcement

The Law of E↵ect requires a backward e↵ect on connections, and some early critics of thelaw could not conceive of how the present could a↵ect something that was in the past. This
concern was amplified by the fact that learning can even occur when there is a considerable
delay between an action and the consequent reward or penalty. Similarly, in classicalconditioning, learning can occur when US onset follows CS o↵set by a non-negligible timeinterval. We call this the problem of delayed reinforcement, which is related to what
Minsky (1961) called the “credit-assignment problem for learning systems”: how do you362

Chapter 14: Psychology

distribute credit for success among the many decisions that may have been involved in
producing it? The reinforcement learning algorithms presented in this book include twobasic mechanisms for addressing this problem. The first is the use of eligibility traces,
and the second is the use of TD methods to learn value functions that provide nearly
immediate evaluations of actions (in tasks like instrumental conditioning experiments) orthat provide immediate prediction targets (in tasks like classical conditioning experiments).
Both of these methods correspond to similar mechanisms proposed in theories of animallearning.
Pavlov (1927) pointed out that every stimulus must leave a trace in the nervous systemthat persists for some time after the stimulus ends, and he proposed that stimulus traces
make learning possible when there is a temporal gap between the CS o↵set and the
US onset. To this day, conditioning under these conditions is called trace conditioning(page 344). Assuming a trace of the CS remains when the US arrives, learning occurs
through the simultaneous presence of the trace and the US. We discuss some proposals
for trace mechanisms in the nervous system in Chapter 15.
Stimulus traces were also proposed as a means for bridging the time interval between
actions and consequent rewards or penalties in instrumental conditioning. In Hull’s
influential learning theory, for example, “molar stimulus traces” accounted for what
he called an animal’s goal gradient, a description of how the maximum strength of an
instrumentally-conditioned response decreases with increasing delay of reinforcement(Hull, 1932, 1943). Hull hypothesized that an animal’s actions leave internal stimuli whose
traces decay exponentially as functions of time since an action was taken. Looking at the
animal learning data available at the time, he hypothesized that the traces e↵ectively
reach zero after 30 to 40 seconds.
The eligibility traces used in the algorithms described in this book are like Hull’s
traces: they are decaying traces of past state visitations, or of past state–action pairs.
Eligibility traces were introduced by Klopf (1972) in his neuronal theory in which they
are temporally-extended traces of past activity at synapses, the connections between
neurons. Klopf’s traces are more complex than the exponentially-decaying traces ouralgorithms use, and we discuss this more when we take up his theory in Section 15.9.
To account for goal gradients that extend over longer time periods than spannedby stimulus traces, Hull (1943) proposed that longer gradients result from conditioned
reinforcement passing backwards from the goal, a process acting in conjunction withhis molar stimulus traces. Animal experiments showed that if conditions favor thedevelopment of conditioned reinforcement during a delay period, learning does notdecrease with increased delay as much as it does under conditions that obstruct secondary
reinforcement. Conditioned reinforcement is favored if there are stimuli that regularly
occur during the delay interval. Then it is as if reward is not actually delayed because
there is more immediate conditioned reinforcement. Hull therefore envisioned that there
is a primary gradient based on the delay of the primary reinforcement mediated bystimulus traces, and that this is progressively modified, and lengthened, by conditioned
reinforcement.
Algorithms presented in this book that use both eligibility traces and value functions
to enable learning with delayed reinforcement correspond to Hull’s hypothesis about how
animals are able to learn under these conditions. The actor–critic architecture discussed

14.5. Cognitive Maps

363

in Sections 13.5, 15.7, and 15.8 illustrates this correspondence most clearly. The critic uses
a TD algorithm to learn a value function associated with the system’s current behavior,that is, to predict the current policy’s return. The actor updates the current policy basedon the critic’s predictions, or more exactly, on changes in the critic’s predictions. The
TD error produced by the critic acts as a conditioned reinforcement signal for the actor,
providing an immediate evaluation of performance even when the primary reward signal
itself is considerably delayed. Algorithms that estimate action-value functions, such as
Q-learning and Sarsa, similarly use TD learning principles to enable learning with delayed
reinforcement by means of conditioned reinforcement. The close parallel between TD
learning and the activity of dopamine producing neurons that we discuss in Chapter 15
lends additional support to links between reinforcement learning algorithms and this
aspect of Hull’s learning theory.

14.5

Cognitive Maps

Model-based reinforcement learning algorithms use environment models that have elements
in common with what psychologists call cognitive maps. Recall from our discussion ofplanning and learning in Chapter 8 that by an environment model we mean anything
an agent can use to predict how its environment will respond to its actions in terms ofstate transitions and rewards, and by planning we mean any process that computes apolicy from such a model. Environment models consist of two parts: the state-transition
part encodes knowledge about the e↵ect of actions on state changes, and the rewardmodel part encodes knowledge about the reward signals expected for each state or eachstate–action pair. A model-based algorithm selects actions by using a model to predictthe consequences of possible courses of action in terms of future states and the reward
signals expected to arise from those states. The simplest kind of planning is to compare
the predicted consequences of collections of “imagined” sequences of decisions.
Questions about whether or not animals use environment models, and if so, what are themodels like and how are they learned, have played influential roles in the history of animallearning research. Some researchers challenged the then-prevailing stimulus-response(S–R) view of learning and behavior, which corresponds to the simplest model-free wayof learning policies, by demonstrating latent learning. In the earliest latent learning
experiment, two groups of rats were run in a maze. For the experimental group, there
was no reward during the first stage of the experiment, but food was suddenly introduced
into the goal box of the maze at the start of the second stage. For the control group, food
was in the goal box throughout both stages. The question was whether or not rats in theexperimental group would have learned anything during the first stage in the absenceof food reward. Although the experimental rats did not appear to learn much during
the first, unrewarded, stage, as soon as they discovered the food that was introducedin the second stage, they rapidly caught up with the rats in the control group. It was
concluded that “during the non-reward period, the rats [in the experimental group] weredeveloping a latent learning of the maze which they were able to utilize as soon as reward
was introduced” (Blodgett, 1929).

364

Chapter 14: Psychology

Latent learning is most closely associated with the psychologist Edward Tolman, whointerpreted this result, and others like it, as showing that animals could learn a “cognitive
map of the environment” in the absence of rewards or penalties, and that they could use
the map later when they were motivated to reach a goal (Tolman, 1948). A cognitive map
could also allow a rat to plan a route to the goal that was di↵erent from the route the rat
had used in its initial exploration. Explanations of results like these led to the enduringcontroversy lying at the heart of the behaviorist/cognitive dichotomy in psychology. In
modern terms, cognitive maps are not restricted to models of spatial layouts but are
more generally environment models, or models of an animal’s “task space” (e.g., Wilson,
Takahashi, Schoenbaum, and Niv, 2014). The cognitive map explanation of latent learning
experiments is analogous to the claim that animals use model-based algorithms, and thatenvironment models can be learned even without explicit rewards or penalties. Models
are then used for planning when the animal is motivated by the appearance of rewards or
penalties.
Tolman’s account of how animals learn cognitive maps was that they learn stimulusstimulus, or S–S, associations by experiencing successions of stimuli as they explore an
environment. In psychology this is called expectancy theory: given S–S associations, theoccurrence of a stimulus generates an expectation about the stimulus to come next. Thisis much like what control engineers call system identification, in which a model of asystem with unknown dynamics is learned from labeled training examples. In the simplest
discrete-time versions, training examples are S–S0 pairs, where S is a state and S0 , thesubsequent state, is the label. When S is observed, the model creates the “expectation”
that S0 will be observed next. Models more useful for planning involve actions as well,so that examples look like SA–S0 , where S0 is expected when action A is executed instate S. It is also useful to learn how the environment generates rewards. In this case,examples are of the form S–R or SA–R, where R is a reward signal associated with S or
the SA pair. These are all forms of supervised learning by which an agent can acquirecognitive-like maps whether or not it receives any non-zero reward signals while exploring
its environment.

14.6

Habitual and Goal-directed Behavior

The distinction between model-free and model-based reinforcement learning algorithms
corresponds to the distinction psychologists make between habitual and goal-directed
control of learned behavioral patterns. Habits are behavior patterns triggered by appropriate stimuli and then performed more-or-less automatically. Goal-directed behavior,according to how psychologists use the phrase, is purposeful in the sense that it is controlled by knowledge of the value of goals and the relationship between actions and theirconsequences. Habits are sometimes said to be controlled by antecedent stimuli, whereas
goal-directed behavior is said to be controlled by its consequences (Dickinson, 1980,1985). Goal-directed control has the advantage that it can rapidly change an animal’s
behavior when the environment changes its way of reacting to the animal’s actions. While
habitual behavior responds quickly to input from an accustomed environment, it is unable14.6. Habitual and Goal-directed Behavior

365

to quickly adjust to changes in the environment. The development of goal-directedbehavioral control was likely a major advance in the evolution of animal intelligence.
Figure 14.5 illustrates the di↵erence between model-free and model-based decision
strategies in a hypothetical task in which a rat has to navigate a maze that has distinctive
goal boxes, each delivering an associated reward of the magnitude shown (Figure 14.5
top). Starting at S1 , the rat has to first select left (L) or right (R) and then has to selectL or R again at S2 or S3 to reach one of the goal boxes. The goal boxes are the terminal
states of each episode of the rat’s episodic task. A model-free strategy (Figure 14.5 lowerleft) relies on stored values for state–action pairs. These action values are estimates of
the highest return the rat can expect for each action taken from each (nonterminal) state.
They are obtained over many trials of running the maze from start to finish. When theaction values have become good enough estimates of the optimal returns, the rat just has

0

S

(4)
4

2

S3

2

3

S1

Reward

S ,L

0

S ,R

4

2

S ,L
1

S ,R
1

4
3

2

L

S

2

S

1

S ,L

2

S ,R

3

3

3

Model-Free

R

S

L

=0

R

=4

L

=2

R

=3

3

Model-Based

Figure 14.5: Model-based and model-free strategies to solve a hypothetical sequential actionselection problem. Top: a rat navigates a maze with distinctive goal boxes, each associatedwith a reward having the value shown. Lower left: a model-free strategy relies on stored action
values for all the state–action pairs obtained over many learning trials. To make decisions therat just has to select at each state the action with the largest action value for that state. Lowerright: in a model-based strategy, the rat learns an environment model, consisting of knowledgeof state–action-next-state transitions and a reward model consisting of knowledge of the reward
associated with each distinctive goal box. The rat can decide which way to turn at each state
by using the model to simulate sequences of action choices to find a path yielding the highest
return. Adapted from Trends in Cognitive Science, volume 10, number 8, Y. Niv, D. Joel, and P.
Dayan, A Normative Perspective on Motivation, p. 376, 2006, with permission from Elsevier.

366

Chapter 14: Psychology

to select at each state the action with the largest action value in order to make optimal
decisions. In this case, when the action-value estimates become accurate enough, therat selects L from S1 and R from S2 to obtain the maximum return of 4. A di↵erent
model-free strategy might simply rely on a cached policy instead of action values, making
direct links from S1 to L and from S2 to R. In neither of these strategies do decisions
rely on an environment model. There is no need to consult a state-transition model, andno connection is required between the features of the goal boxes and the rewards they
deliver.
Figure 14.5 (lower right) illustrates a model-based strategy. It uses an environment
model consisting of a state-transition model and a reward model. The state-transition
model is shown as a decision tree, and the reward model associates the distinctive featuresof the goal boxes with the rewards to be found in each. (The rewards associated withstates S1 , S2 , and S3 are also part of the reward model, but here they are zero and are
not shown.) A model-based agent can decide which way to turn at each state by using
the model to simulate sequences of action choices to find a path yielding the highest
return. In this case the return is the reward obtained from the outcome at the end of
the path. Here, with a sufficiently accurate model, the rat would select L and then R
to obtain a return of 4. Comparing the predicted returns of simulated paths is a simple
form of planning, which can be done in a variety of ways as discussed in Chapter 8.
When the environment of a model-free agent changes the way it reacts to the agent’s
actions, the agent has to acquire new experience in the changed environment during
which it can update its policy and/or value function. In the model-free strategy shownin Figure 14.5 (lower left), for example, if one of the goal boxes were to somehow shift
to delivering a di↵erent reward, the rat would have to traverse the maze, possibly manytimes, to experience the new reward upon reaching that goal box, all the while updating
either its policy or its action-value function (or both) based on this experience. The key
point is that for a model-free agent to change the action its policy specifies for a state, or
to change an action value associated with a state, it has to move to that state, act fromit, possibly many times, and experience the consequences of its actions.
A model-based agent can accommodate changes in its environment without this kindof ‘personal experience’ with the states and actions a↵ected by the change. A change inits model automatically (through planning) changes its policy. Planning can determine
the consequences of changes in the environment that have never been linked together in
the agent’s own experience. For example, again referring to the maze task of Figure 14.5,
imagine that a rat with a previously learned transition and reward model is placed directlyin the goal box to the right of S2 to find that the reward available there now has value 1
instead of 4. The rat’s reward model will change even though the action choices required
to find that goal box in the maze were not involved. The planning process will bringknowledge of the new reward to bear on maze running without the need for additional
experience in the maze; in this case changing the policy to right turns at both S1 and S3
to obtain a return of 3.
Exactly this logic is the basis of outcome-devaluation experiments with animals. Resultsfrom these experiments provide insight into whether an animal has learned a habit or if
its behavior is under goal-directed control. Outcome-devaluation experiments are likelatent-learning experiments in that the reward changes from one stage to the next. After

14.6. Habitual and Goal-directed Behavior

367

an initial rewarded stage of learning, the reward value of an outcome is changed, including
being shifted to zero or even to a negative value.
An early important experiment of this type was conducted by Adams and Dickinson(1981). They trained rats via instrumental conditioning until the rats energetically presseda lever for sucrose pellets in a training chamber. The rats were then placed in the samechamber with the lever retracted and allowed non-contingent food, meaning that pellets
were made available to them independently of their actions. After 15-minutes of thisfree-access to the pellets, rats in one group were injected with the nausea-inducing poison
lithium chloride. This was repeated for three sessions, in the last of which none of theinjected rats consumed any of the non-contingent pellets, indicating that the reward
value of the pellets had been decreased—the pellets had been devalued. In the next stage
taking place a day later, the rats were again placed in the chamber and given a session of
extinction training, meaning that the response lever was back in place but disconnectedfrom the pellet dispenser so that pressing it did not release pellets. The question waswhether the rats that had the reward value of the pellets decreased would lever-press
less than rats that did not have the reward value of the pellets decreased, even without
experiencing the devalued reward as a result of lever-pressing. It turned out that the
injected rats had significantly lower response rates than the non-injected rats right fromthe start of the extinction trials.
Adams and Dickinson concluded that the injected rats associated lever pressing withconsequent nausea by means of a cognitive map linking lever pressing to pellets, andpellets to nausea. Hence, in the extinction trials, the rats “knew” that the consequencesof pressing the lever would be something they did not want, and so they reduced theirlever-pressing right from the start. The important point is that they reduced lever-pressing
without ever having experienced lever-pressing directly followed by being sick: no lever
was present when they were made sick. They seemed able to combine knowledge of theoutcome of a behavioral choice (pressing the lever will be followed by getting a pellet)
with the reward value of the outcome (pellets are to be avoided) and hence could alter
their behavior accordingly. Not every psychologist agrees with this “cognitive” accountof this kind of experiment, and it is not the only possible way to explain these results,
but the model-based planning explanation is widely accepted.
Nothing prevents an agent from using both model-free and model-based algorithms, andthere are good reasons for using both. We know from our own experience that with enoughrepetition, goal-directed behavior tends to turn into habitual behavior. Experiments showthat this happens for rats too. Adams (1982) conducted an experiment to see if extended
training would convert goal-directed behavior into habitual behavior. He did this bycomparing the e↵ect of outcome devaluation on rats that experienced di↵erent amountsof training. If extended training made the rats less sensitive to devaluation compared torats that received less training, this would be evidence that extended training made thebehavior more habitual. Adams’ experiment closely followed the Adams and Dickinson(1981) experiment just described. Simplifying a bit, rats in one group were trained until
they made 100 rewarded lever-presses, and rats in the other group—the overtrainedgroup—were trained until they made 500 rewarded lever-presses. After this training,
the reward value of the pellets was decreased (using lithium chloride injections) for rats

368

Chapter 14: Psychology

in both groups. Then both groups of rats were given a session of extinction training.
Adams’ question was whether devaluation would e↵ect the rate of lever-pressing for theovertrained rats less than it would for the non-overtrained rats, which would be evidence
that extended training reduces sensitivity to outcome devaluation. It turned out thatdevaluation strongly decreased the lever-pressing rate of the non-overtrained rats. For
the overtrained rats, in contrast, devaluation had little e↵ect on their lever-pressing; infact, if anything, it made it more vigorous. (The full experiment included control groups
showing that the di↵erent amounts of training did not by themselves significantly e↵ectlever-pressing rates after learning.) This result suggested that while the non-overtrained
rats were acting in a goal-directed manner sensitive to their knowledge of the outcome of
their actions, the overtrained rats had developed a lever-pressing habit.
Viewing this and other results like it from a computational perspective provides insightas to why one might expect animals to behave habitually in some circumstances but in agoal-directed way in others, and why they shift from one mode of control to another as
they continue to learn. While animals undoubtedly use algorithms that do not exactly
match those we have presented in this book, one can gain insight into animal behavior byconsidering the tradeo↵s that various reinforcement learning algorithms imply. An idea
developed by computational neuroscientists Daw, Niv, and Dayan (2005) is that animals
use both model-free and model-based processes. Each process proposes an action, andthe action chosen for execution is the one proposed by the process judged to be the moretrustworthy of the two as determined by measures of confidence that are maintained
throughout learning. Early in learning the planning process of a model-based system ismore trustworthy because it chains together short-term predictions which can become
accurate with less experience than long-term predictions of the model-free process. But
with continued experience, the model-free process becomes more trustworthy because
planning is prone to making mistakes due to model inaccuracies and short-cuts necessary
to make planning feasible, such as various forms of “tree-pruning”: the removal of
unpromising search tree branches. According to this idea one would expect a shift fromgoal-directed behavior to habitual behavior as more experience accumulates. Other ideas
have been proposed for how animals arbitrate between goal-directed and habitual control,and both behavioral and neuroscience research continues to examine this and related
questions.
The distinction between model-free and model-based algorithms is proving to be usefulfor this research. One can examine the computational implications of these types ofalgorithms in abstract settings that expose basic advantages and limitations of each
type. This serves both to suggest and to sharpen questions that guide the designof experiments necessary for increasing psychologists’ understanding of habitual andgoal-directed behavioral control.

14.7

Summary

Our goal in this chapter has been to discuss correspondences between reinforcement
learning and the experimental study of animal learning in psychology. We emphasized
at the outset that reinforcement learning as described in this book is not intended

14.7. Summary369

to model details of animal behavior. It is an abstract computational framework that
explores idealized situations from the perspective of artificial intelligence and engineering.
But many of the basic reinforcement learning algorithms were inspired by psychologicaltheories, and in some cases, these algorithms have contributed to the development of
new animal learning models. This chapter described the most conspicuous of these
correspondences.
The distinction in reinforcement learning between algorithms for prediction and algorithms for control parallels animal learning theory’s distinction between classical, orPavlovian, conditioning and instrumental conditioning. The key di↵erence between
instrumental and classical conditioning experiments is that in the former the reinforcing
stimulus is contingent upon the animal’s behavior, whereas in the latter it is not. Learning
to predict via a TD algorithm corresponds to classical conditioning, and we described
the TD model of classical conditioning as one instance in which reinforcement learning
principles account for some details of animal learning behavior. This model generalizes
the influential Rescorla–Wagner model by including the temporal dimension where events
within individual trials influence learning, and it provides an account of second-orderconditioning, where predictors of reinforcing stimuli become reinforcing themselves. It
also is the basis of an influential view of the activity of dopamine neurons in the brain,something we take up in Chapter 15.
Learning by trial and error is at the base of the control aspect of reinforcement learning.
We presented some details about Thorndike’s experiments with cats and other animals that
led to his Law of E↵ect, which we discussed here and in Chapter 1 (page 15). We pointed
out that in reinforcement learning, exploration does not have to be limited to “blind
groping”; trials can be generated by sophisticated methods using innate and previously
learned knowledge as long as there is some exploration. We discussed the training method
B. F. Skinner called shaping in which reward contingencies are progressively altered totrain an animal to successively approximate a desired behavior. Shaping is not only
indispensable for animal training, it is also an e↵ective tool for training reinforcement
learning agents. There is also a connection to the idea of an animal’s motivational state,
which influences what an animal will approach or avoid and what events are rewardingor punishing for the animal.
The reinforcement learning algorithms presented in this book include two basic mechanisms for addressing the problem of delayed reinforcement: eligibility traces and valuefunctions learned via TD algorithms. Both mechanisms have antecedents in theories ofanimal learning. Eligibility traces are similar to stimulus traces of early theories, andvalue functions correspond to the role of secondary reinforcement in providing nearly
immediate evaluative feedback.
The next correspondence the chapter addressed is that between reinforcement learning’s
environment models and what psychologists call cognitive maps. Experiments conductedin the mid 20th century purported to demonstrate the ability of animals to learn cognitive
maps as alternatives to, or as additions to, state–action associations, and later use themto guide behavior, especially when the environment changes unexpectedly. Environment
models in reinforcement learning are like cognitive maps in that they can be learned bysupervised learning methods without relying on reward signals, and then they can beused later to plan behavior.

370

Chapter 14: Psychology

Reinforcement learning’s distinction between model-free and model-based algorithms
corresponds to the distinction in psychology between habitual and goal-directed behavior.
Model-free algorithms make decisions by accessing information that has been stored in apolicy or an action-value function, whereas model-based methods select actions as theresult of planning ahead using a model of the agent’s environment. Outcome-devaluation
experiments provide information about whether an animal’s behavior is habitual or undergoal-directed control. Reinforcement learning theory has helped clarify thinking about
these issues.
Animal learning clearly informs reinforcement learning, but as a type of machinelearning, reinforcement learning is directed toward designing and understanding e↵ective
learning algorithms, not toward replicating or explaining details of animal behavior.
We focused on aspects of animal learning that relate in clear ways to methods forsolving prediction and control problems, highlighting the fruitful two-way flow of ideas
between reinforcement learning and psychology without venturing deeply into many of the
behavioral details and controversies that have occupied the attention of animal learning
researchers. Future development of reinforcement learning theory and algorithms will
likely exploit links to many other features of animal learning as the computational utilityof these features becomes better appreciated. We expect that a flow of ideas between
reinforcement learning and psychology will continue to bear fruit for both disciplines.
Many connections between reinforcement learning and areas of psychology and otherbehavioral sciences are beyond the scope of this chapter. We largely omit discussing
links to the psychology of decision making, which focuses on how actions are selected,
or how decisions are made, after learning has taken place. We also do not discuss links
to ecological and evolutionary aspects of behavior studied by ethologists and behavioral
ecologists: how animals relate to one another and to their physical surroundings, and how
their behavior contributes to evolutionary fitness. Optimization, MDPs, and dynamic
programming figure prominently in these fields, and our emphasis on agent interaction
with dynamic environments connects to the study of agent behavior in complex “ecologies.”
Multi-agent reinforcement learning, omitted in this book, has connections to social aspectsof behavior. Despite the lack of treatment here, reinforcement learning should by no means
be interpreted as dismissing evolutionary perspectives. Nothing about reinforcement
learning implies a tabula rasa view of learning and behavior. Indeed, experience withengineering applications has highlighted the importance of building into reinforcement
learning systems knowledge that is analogous to what evolution provides to animals.

14.7. Summary

371

Bibliographical and Historical Remarks
Ludvig, Bellemare, and Pearson (2011) and Shah (2012) review reinforcement learning in
the contexts of psychology and neuroscience. These publications are useful companions
to this chapter and the following chapter on reinforcement learning and neuroscience.
14.1

Dayan, Niv, Seymour, and Daw (2006) focused on interactions between classical and instrumental conditioning, particularly situations where classicallyconditioned and instrumental responses are in conflict. They proposed a Qlearning framework for modeling aspects of this interaction. Modayil and Sutton(2014) used a mobile robot to demonstrate the e↵ectiveness of a control method
combining a fixed response with online prediction learning. Calling this Pavlovian control , they emphasized that it di↵ers from the usual control methods ofreinforcement learning, being based on predictively executing fixed responses
and not on reward maximization. The electro-mechanical machine of Ross (1933)and especially the learning version of Walter’s turtle (Walter, 1951) were very
early illustrations of Pavlovian control.

14.2.1 Kamin (1968) first reported blocking, now commonly known as Kamin blocking,in classical conditioning. Moore and Schmajuk (2008) provide an excellent
summary of the blocking phenomenon, the research it stimulated, and its lasting
influence on animal learning theory. Gibbs, Cool, Land, Kehoe, and Gormezano(1991) describe second-order conditioning of the rabbit’s nictitating membrane
response and its relationship to conditioning with serial-compound stimuli. Finchand Culler (1934) reported obtaining fifth-order conditioning of a dog’s foreleg
withdrawal “when the motivation of the animal is maintained through the various
orders.”14.2.2 The idea built into the Rescorla–Wagner model that learning occurs when animals
are surprised is derived from Kamin (1969). Models of classical conditioning
other than Rescorla and Wagner’s include the models of Klopf (1988), Grossberg(1975), Mackintosh (1975), Moore and Stickney (1980), Pearce and Hall (1980),
and Courville, Daw, and Touretzky (2006). Schmajuk (2008) reviews models ofclassical conditioning. Wagner (2008) provides a modern psychological perspectiveon the Rescorla-Wagner model and similar elemental theories of learning.
14.2.3 An early version of the TD model of classical conditioning appeared in Sutton andBarto (1981a), which also included the early model’s prediction that temporal
primacy overrides blocking, later shown by Kehoe, Schreurs, and Graham (1987)
to occur in the rabbit nictitating membrane preparation. Sutton and Barto(1981a) contains the earliest recognition of the near identity between the Rescorla–
Wagner model and the Least-Mean-Square (LMS), or Widrow-Ho↵, learning
rule (Widrow and Ho↵, 1960). This early model was revised following Sutton’sdevelopment of the TD algorithm (Sutton, 1984, 1988) and was first presented as
the TD model in Sutton and Barto (1987) and more completely in Sutton andBarto (1990), upon which this section is largely based. Additional exploration

372

Chapter 14: Psychologyof the TD model and its possible neural implementation was conducted by
Moore and colleagues (Moore, Desmond, Berthier, Blazis, Sutton, and Barto,
1986; Moore and Blazis, 1989; Moore, Choi, and Brunzell, 1998; Moore, Marks,
Castagna, and Polewan, 2001). Klopf’s (1988) drive-reinforcement theory ofclassical conditioning extends the TD model to address additional experimental
details, such as the S-shape of acquisition curves. In some of these publications
TD is taken to mean Time Derivative instead of Temporal Di↵erence.
14.2.4 Ludvig, Sutton, and Kehoe (2012) evaluated the performance of the TD modelin previously unexplored tasks involving classical conditioning and examined
the influence of various stimulus representations, including the microstimulus
representation that they introduced earlier (Ludvig, Sutton, and Kehoe, 2008).
Earlier investigations of the influence of various stimulus representations and their
possible neural implementations on response timing and topography in the context
of the TD model are those of Moore and colleagues cited above. Although not in
the context of the TD model, representations like the microstimulus representationof Ludvig et al. (2012) have been proposed and studied by Grossberg andSchmajuk (1989), Brown, Bullock, and Grossberg (1999), Buhusi and Schmajuk(1999), and Machado (1997). The figures on pages 353–355 are adapted from
Sutton and Barto (1990).
14.3

Section 1.7 includes comments on the history of trial-and-error learning andthe Law of E↵ect. The idea that Thorndike’s cats might have been exploring
according to an instinctual context-specific ordering over actions rather than byjust selecting from a set of instinctual impulses was suggested by Peter Dayan(personal communication). Selfridge, Sutton, and Barto (1985) illustrated thee↵ectiveness of shaping in a pole-balancing reinforcement learning task. Other
examples of shaping in reinforcement learning are Gullapalli and Barto (1992),
Mahadevan and Connell (1992), Mataric (1994), Dorigo and Colombette (1994),
Saksida, Raymond, and Touretzky (1997), and Randløv and Alstrøm (1998). Ng(2003) and Ng, Harada, and Russell (1999) used the term shaping in a sense
somewhat di↵erent from Skinner’s, focusing on the problem of how to alter the
reward signal without altering the set of optimal policies.
Dickinson and Balleine (2002) discuss the complexity of the interaction between
learning and motivation. Wise (2004) provides an overview of reinforcement
learning and its relation to motivation. Daw and Shohamy (2008) link motivationand learning to aspects of reinforcement learning theory. See also McClure,
Daw, and Montague (2003), Niv, Joel, and Dayan (2006), Rangel, Camerer, andMontague (2008), and Dayan and Berridge (2014). McClure et al. (2003), Niv,
Daw, and Dayan (2006), and Niv, Daw, Joel, and Dayan (2007) present theories
of behavioral vigor related to the reinforcement learning framework.

14.4

Spence, Hull’s student and collaborator at Yale, elaborated the role of higherorder reinforcement in addressing the problem of delayed reinforcement (Spence,1947). Learning over very long delays, as in taste-aversion conditioning with

14.7. Summary

373

delays up to several hours, led to interference theories as alternatives to decayingtrace theories (e.g., Revusky and Garcia, 1970; Boakes and Costa, 2014). Other
views of learning under delayed reinforcement invoke roles for awareness andworking memory (e.g., Clark and Squire, 1998; Seo, Barraclough, and Lee, 2007).
14.5

Thistlethwaite (1951) provides an extensive review of latent learning experiments
up to the time of its publication. Ljung (1998) provides an overview of model
learning, or system identification, techniques in engineering. Gopnik, Glymour,
Sobel, Schulz, Kushnir, and Danks (2004) present a Bayesian theory about how
children learn models.

14.6

Connections between habitual and goal-directed behavior and model-free andmodel-based reinforcement learning were first proposed by Daw, Niv, and Dayan(2005). The hypothetical maze task used to explain habitual and goal-directedbehavioral control is based on the explanation of Niv, Joel, and Dayan (2006).
Dolan and Dayan (2013) review four generations of experimental research related
to this issue and discuss how it can move forward on the basis of reinforcement
learning’s model-free/model-based distinction. Dickinson (1980, 1985) and Dickinson and Balleine (2002) discuss experimental evidence related to this distinction.
Donahoe and Burgos (2000) alternatively argue that model-free processes canaccount for the results of outcome-devaluation experiments. Dayan and Berridge(2014) argue that classical conditioning involves model-based processes. Rangel,
Camerer, and Montague (2008) review many of the outstanding issues involving
habitual, goal-directed, and Pavlovian modes of control.

Comments on Terminology— The traditional meaning of reinforcement in psychology is the strengthening of a pattern of behavior (by increasing either its intensity or
frequency) as a result of an animal receiving a stimulus (or experiencing the omissionof a stimulus) in an appropriate temporal relationship with another stimulus or with a
response. Reinforcement produces changes that remain in future behavior. Sometimes in
psychology reinforcement refers to the process of producing lasting changes in behavior,
whether the changes strengthen or weaken a behavior pattern (Mackintosh, 1983). Letting
reinforcement refer to weakening in addition to strengthening is at odds with the everyday
meaning of reinforce, and its traditional use in psychology, but it is a useful extensionthat we have adopted here. In either case, a stimulus considered to be the cause of the
behavioral change is called a reinforcer.
Psychologists do not generally use the specific phrase reinforcement learning as we
do. Animal learning pioneers probably regarded reinforcement and learning as being
synonymous, so it would be redundant to use both words. Our use of the phrase follows
its use in computational and engineering research, influenced mostly by Minsky (1961).
But the phrase is lately gaining currency in psychology and neuroscience, likely because
strong parallels have surfaced between reinforcement learning algorithms and animallearning—parallels described in this chapter and the next.
According to common usage, a reward is an object or event that an animal willapproach and work for. A reward may be given to an animal in recognition of its ‘good’374

Chapter 14: Psychology

behavior, or given in order to make the animal’s behavior ‘better.’ Similarly, a penalty is
an object or event that the animal usually avoids and that is given as a consequence of‘bad’ behavior, usually in order to change that behavior. Primary reward is reward due
to machinery built into an animal’s nervous system by evolution to improve its chancesof survival and reproduction, for example, reward produced by the taste of nourishingfood, sexual contact, successful escape, and many other stimuli and events that predicted
reproductive success over the animal’s ancestral history. As explained in Section 14.2.1,
higher-order reward is reward delivered by stimuli that predict primary reward, either
directly or indirectly by predicting other stimuli that predict primary reward. Reward issecondary if its rewarding quality is the result of directly predicting primary reward.
In this book we call Rt the ‘reward signal at time t,’ or sometimes just the ‘reward
at time t,’ but we do not think of it as an object or event in the agent’s environment.
Because Rt is a number—not an object or an event—it is more like a reward signal inneuroscience, which is a signal internal to the brain, like the activity of neurons, that
influences decision making and learning. This signal might be triggered when the animal
perceives an attractive (or an aversive) object, but it can also be triggered by things that
do not physically exist in the animal’s external environment, such as memories, ideas, or
hallucinations. Because our Rt can be positive, negative, or zero, it might be better tocall a negative Rt a penalty, and an Rt equal to zero a neutral signal, but for simplicity
we generally avoid these terms.
In reinforcement learning, the process that generates all the Rt s defines the problem
the agent is trying to solve. The agent’s objective is to keep the magnitude of Rt as large
as possible over time. In this respect, Rt is like primary reward for an animal if we thinkof the problem the animal faces as the problem of obtaining as much primary reward as
possible over its lifetime (and thereby, through the prospective “wisdom” of evolution,
improve its chances of solving its real problem, which is to pass its genes on to future
generations). However, as we suggest in Chapter 15, it is unlikely that there is a single“master” reward signal like Rt in an animal’s brain.
Not all reinforcers are rewards or penalties. Sometimes reinforcement is not the resultof an animal receiving a stimulus that evaluates its behavior by labeling the behavior
good or bad. A behavior pattern can be reinforced by a stimulus that arrives to an animal
no matter how the animal behaved. As described in Section 14.1, whether the delivery of
reinforcement depends, or does not depend, on preceding behavior is the defining di↵erence
between instrumental, or operant, conditioning experiments and classical, or Pavlovian,conditioning experiments. Reinforcement is at work in both types of experiments, but
only in the former is it feedback that evaluates past behavior. (Though it has often been
pointed out that even when the reinforcing US in a classical conditioning experiment is
not contingent on the subject’s preceding behavior, its reinforcing value can be influenced
by this behavior, an example being that a closed eye makes an air pu↵ to the eye lessaversive.)
The distinction between reward signals and reinforcement signals is a crucial point
when we discuss neural correlates of these signals in the next chapter. Like a reward signal,
for us, the reinforcement signal at any specific time is a positive or negative number, orzero. A reinforcement signal is the major factor directing changes a learning algorithm

14.7. Summary

375

makes in an agent’s policy, value estimates, or environment models. The definition thatmakes the most sense to us is that a reinforcement signal at any time is a number thatmultiplies (possibly along with some constants) a vector to determine parameter updates
in some learning algorithm.
For some algorithms, the reward signal alone is the critical multiplier in the parameterupdate equation. For these algorithms the reinforcement signal is the same as the
reward signal. But for most of the algorithms we discuss in this book, reinforcement
signals include terms in addition to the reward signal, an example being a TD error
V (St ), which is the reinforcement signal for TD state-valuet = Rt+1 + V (St+1 )learning (and analogous TD errors for action-value learning). In this reinforcement signal,
Rt+1 is the primary reinforcement contribution, and the temporal di↵erence in predictedvalues, V (St+1 ) V (St ) (or an analogous temporal di↵erence for action values), is
the conditioned reinforcement contribution. Thus, whenever V (St+1 ) V (St ) = 0, t
signals ‘pure’ primary reinforcement; and whenever Rt+1 = 0, it signals ‘pure’ conditioned
reinforcement, but it often signals a mixture of these. Note as we mentioned in Section 6.1,
this t is not available until time t + 1. We therefore think of t as the reinforcement
signal at time t + 1, which is fitting because it reinforces predictions and/or actions madeearlier at step t.
A possible source of confusion is the terminology used by the famous psychologist
B. F. Skinner and his followers. For Skinner, positive reinforcement occurs when theconsequences of an animal’s behavior increase the frequency of that behavior; punishment
occurs when the behavior’s consequences decrease that behavior’s frequency. Negative
reinforcement occurs when behavior leads to the removal of an aversive stimulus (that is,a stimulus the animal does not like), thereby increasing the frequency of that behavior.
Negative punishment, on the other hand, occurs when behavior leads to the removal of anappetitive stimulus (that is, a stimulus the animal likes), thereby decreasing the frequencyof that behavior. We find no critical need for these distinctions because our approach
is more abstract than this, with both reward and reinforcement signals allowed to takeon both positive and negative values. (But note especially that when our reinforcement
signal is negative, it is not the same as Skinner’s negative reinforcement.)
On the other hand, it has often been pointed out that using a single number as areward or a penalty signal, depending only on its sign, is at odds with the fact thatanimals’ appetitive and aversive systems have qualitatively di↵erent properties andinvolve di↵erent brain mechanisms. This points to a direction in which the reinforcement
learning framework might be developed in the future to exploit computational advantagesof separate appetitive and aversive systems, but for now we are passing over these
possibilities.
Another discrepancy in terminology is how we use the word action. To many cognitive
scientists, an action is purposeful in the sense of being the result of an animal’s knowledge
about the relationship between the behavior in question and the consequences of thatbehavior. An action is goal-directed and the result of a decision, whereas a response
is triggered by a stimulus and is the result of a reflex or a habit. We use the word
action without di↵erentiating among what others call actions, decisions, and responses.
These are important distinctions, but for us they are encompassed by di↵erences between

376

Chapter 14: Psychology

model-free and model-based reinforcement learning algorithms, which we discussed above
in relation to habitual and goal-directed behavior in Section 14.6. Dickinson (1985)
discusses the distinction between responses and actions.
A term used a lot in this book is control. What we mean by control is entirely di↵erent
from what it means to animal learning psychologists. By control we mean that an agent
influences its environment to bring about states or events that the agent prefers: the agent
exerts control over its environment. This is the sense of control used by control engineers.
In psychology, on the other hand, control typically means that an animal’s behavior is
influenced by—is controlled by—the stimuli the animal receives (stimulus control) or
the reinforcement schedule it experiences. Here the environment is controlling the agent.
Control in this sense is the basis of behavior modification therapy. Of course, both ofthese directions of control are at play when an agent interacts with its environment,but our focus is on the agent as controller, not the environment as controller. A viewequivalent to ours, and perhaps more illuminating, is that the agent is actually controllingthe input it receives from its environment (Powers, 1973). This is not what psychologists
mean by stimulus control.
Sometimes reinforcement learning is understood to refer solely to learning policies
directly from rewards (and penalties) without the involvement of value functions orenvironment models. This is what psychologists call stimulus-response, or S-R, learning.
But for us, along with most of today’s psychologists, reinforcement learning is muchbroader than this, including in addition to S-R learning, methods involving value functions,environment models, planning, and other processes that are commonly thought to belong
to the more cognitive side of mental functioning.

Chapter 15

Neuroscience
Neuroscience is the multidisciplinary study of nervous systems: how they regulate bodilyfunctions; control behavior; change over time as a result of development, learning, andaging; and how cellular and molecular mechanisms make these functions possible. Oneof the most exciting aspects of reinforcement learning is the mounting evidence fromneuroscience that the nervous systems of humans and many other animals implement
algorithms that correspond in striking ways to reinforcement learning algorithms. The
main objective of this chapter is to explain these parallels and what they suggest about
the neural basis of reward-related learning in animals.
The most remarkable point of contact between reinforcement learning and neuroscience
involves dopamine, a chemical deeply involved in reward processing in the brains of mammals. Dopamine appears to convey temporal-di↵erence (TD) errors to brain structures
where learning and decision making take place. This parallel is expressed by the reward
prediction error hypothesis of dopamine neuron activity, a hypothesis that resulted fromthe convergence of computational reinforcement learning and results of neuroscience
experiments. In this chapter we discuss this hypothesis, the neuroscience findings that
led to it, and why it is a significant contribution to understanding brain reward systems.
We also discuss parallels between reinforcement learning and neuroscience that are lessstriking than this dopamine/TD-error parallel but that provide useful conceptual tools
for thinking about reward-based learning in animals. Other elements of reinforcement
learning have the potential to impact the study of nervous systems, but their connections
to neuroscience are still relatively undeveloped. We discuss several of these evolving
connections that we think will grow in importance over time.
As we outlined in the history section of this book’s introductory chapter (Section 1.7),
many aspects of reinforcement learning were influenced by neuroscience. A second
objective of this chapter is to acquaint readers with ideas about brain function that havecontributed to our approach to reinforcement learning. Some elements of reinforcement
learning are easier to understand when seen in light of theories of brain function. Thisis particularly true for the idea of the eligibility trace, one of the basic mechanismsof reinforcement learning, that originated as a conjectured property of synapses, thestructures by which nerve cells—neurons—communicate with one another.

378

Chapter 15: Neuroscience

In this chapter we do not delve very deeply into the enormous complexity of the neuralsystems underlying reward-based learning in animals: this chapter is too short, and we are
not neuroscientists. We do not try to describe—or even to name—the very many brain
structures and pathways, or any of the molecular mechanisms, believed to be involved inthese processes. We also do not do justice to hypotheses and models that are alternatives
to those that align so well with reinforcement learning. It should not be surprising that
there are di↵ering views among experts in the field. We can only provide a glimpse into
this fascinating and developing story. We hope, though, that this chapter convinces
you that a very fruitful channel has emerged connecting reinforcement learning and its
theoretical underpinnings to the neuroscience of reward-based learning in animals.
Many excellent publications cover links between reinforcement learning and neuroscience, some of which we cite in this chapter’s final section. Our treatment di↵ers frommost of these because we assume familiarity with reinforcement learning as presented
in the earlier chapters of this book, but we do not assume knowledge of neuroscience.
We begin with a brief introduction to the neuroscience concepts needed for a basic
understanding of what is to follow.

15.1

Neuroscience Basics

Some basic information about nervous systems is helpful for following what we cover inthis chapter. Terms that we refer to later are italicized. Skipping this section will not be
a problem if you already have an elementary knowledge of neuroscience.
Neurons, the main components of nervous systems, are cells specialized for processing
and transmitting information using electrical and chemical signals. They come in many
forms, but a neuron typically has a cell body, dendrites, and a single axon. Dendrites
are structures that branch from the cell body to receive input from other neurons (or toalso receive external signals in the case of sensory neurons). A neuron’s axon is a fiberthat carries the neuron’s output to other neurons (or to muscles or glands). A neuron’s
output consists of sequences of electrical pulses called action potentials that travel along
the axon. Action potentials are also called spikes, and a neuron is said to fire when it
generates a spike. In models of neural networks it is common to use real numbers to
represent a neuron’s firing rate, the average number of spikes per some unit of time.
A neuron’s axon can branch widely so that the neuron’s action potentials reachmany targets. The branching structure of a neuron’s axon is called the neuron’s axonal
arbor. Because the conduction of an action potential is an active process, not unlike theburning of a fuse, when an action potential reaches an axonal branch point it “lights
up” action potentials on all of the outgoing branches (although propagation to a branch
can sometimes fail). As a result, the activity of a neuron with a large axonal arbor can
influence many target sites.

15.1. Neuroscience Basics

379

A synapse is a structure generally at the termination of an axon branch that mediates
the communication of one neuron to another. A synapse transmits information fromthe presynaptic neuron’s axon to a dendrite or cell body of the postsynaptic neuron.
With a few exceptions, synapses release a chemical neurotransmitter upon the arrivalof an action potential from the presynaptic neuron. (The exceptions are cases of directelectric coupling between neurons, but these will not concern us here.) Neurotransmittermolecules released from the presynaptic side of the synapse di↵use across the synaptic
cleft, the very small space between the presynaptic ending and the postsynaptic neuron,and then bind to receptors on the surface of the postsynaptic neuron to excite or inhibit
its spike-generating activity, or to modulate its behavior in other ways. A particular
neurotransmitter may bind to several di↵erent types of receptors, with each producing adi↵erent e↵ect on the postsynaptic neuron. For example, there are at least five di↵erent
receptor types by which the neurotransmitter dopamine can a↵ect a postsynaptic neuron.
Many di↵erent chemicals have been identified as neurotransmitters in animal nervous
systems.
A neuron’s background activity is its level of activity, usually its firing rate, when theneuron does not appear to be driven by synaptic input related to the task of interest
to the experimenter, for example, when the neuron’s activity is not correlated with astimulus delivered to a subject as part of an experiment. Background activity can beirregular due to input from the wider network, or due to noise within the neuron or its
synapses. Sometimes background activity is the result of dynamic processes intrinsic to
the neuron. A neuron’s phasic activity, in contrast to its background activity, consists ofbursts of spiking activity usually caused by synaptic input. Activity that varies slowly
and often in a graded manner, whether as background activity or not, is called a neuron’s
tonic activity.
The strength or e↵ectiveness by which the neurotransmitter released at a synapse
influences the postsynaptic neuron is the synapse’s efficacy. One way a nervous system
can change through experience is through changes in synaptic efficacies as a result ofcombinations of the activities of the presynaptic and postsynaptic neurons, and sometimes
by the presence of a neuromodulator, which is a neurotransmitter having e↵ects otherthan, or in addition to, direct fast excitation or inhibition.
Brains contain several di↵erent neuromodulation systems consisting of clusters ofneurons with widely branching axonal arbors, with each system using a di↵erent neurotransmitter. Neuromodulation can alter the function of neural circuits, mediate motivation,
arousal, attention, memory, mood, emotion, sleep, and body temperature. Important
here is that a neuromodulatory system can distribute something like a scalar signal, suchas a reinforcement signal, to alter the operation of synapses in widely distributed sites
critical for learning.
The ability of synaptic efficacies to change is called synaptic plasticity. It is one of the
primary mechanisms responsible for learning. The parameters, or weights, adjusted bylearning algorithms correspond to synaptic efficacies. As we detail below, modulation ofsynaptic plasticity via the neuromodulator dopamine is a plausible mechanism for how
the brain might implement learning algorithms like many of those described in this book.

380

15.2

Chapter 15: Neuroscience

Reward Signals, Reinforcement Signals, Values,
and Prediction Errors

Links between neuroscience and computational reinforcement learning begin as parallels
between signals in the brain and signals playing prominent roles in reinforcement learningtheory and algorithms. In Chapter 3 we said that any problem of learning goal-directedbehavior can be reduced to the three signals representing actions, states, and rewards.
However, to explain links that have been made between neuroscience and reinforcement
learning, we have to be less abstract than this and consider other reinforcement learning
signals that correspond, in certain ways, to signals in the brain. In addition to reward
signals, these include reinforcement signals (which we argue are di↵erent from reward
signals), value signals, and signals conveying prediction errors. When we label a signal by
its function in this way, we are doing it in the context of reinforcement learning theory
in which the signal corresponds to a term in an equation or an algorithm. On the other
hand, when we refer to a signal in the brain, we mean a physiological event such asa burst of action potentials or the secretion of a neurotransmitter. Labeling a neuralsignal by its function, for example calling the phasic activity of a dopamine neuron areinforcement signal, means that the neural signal behaves like, and is conjectured tofunction like, the corresponding theoretical signal.
Uncovering evidence for these correspondences involves many challenges. Neuralactivity related to reward processing can be found in nearly every part of the brain,
and it is difficult to interpret results unambiguously because representations of di↵erent
reward-related signals tend to be highly correlated with one another. Experiments need tobe carefully designed to allow one type of reward-related signal to be distinguished withany degree of certainty from others—or from an abundance of other signals not related to
reward processing. Despite these difficulties, many experiments have been conducted withthe aim of reconciling aspects of reinforcement learning theory and algorithms with neuralsignals, and some compelling links have been established. To prepare for examining theselinks, in the rest of this section we remind the reader of what various reward-related
signals mean according to reinforcement learning theory.
In our Comments on Terminology at the end of the previous chapter, we said that
Rt is like a reward signal in an animal’s brain and not an object or event in the
animal’s environment. In reinforcement learning, the reward signal (along with an agent’s
environment) defines the problem a reinforcement learning agent is trying to solve. In
this respect, Rt is like a signal in an animal’s brain that distributes primary reward tosites throughout the brain. But it is unlikely that a unitary master reward signal like Rt
exists in an animal’s brain. It is best to think of Rt as an abstraction summarizing theoverall e↵ect of a multitude of neural signals generated by many systems in the brainthat assess the rewarding or punishing qualities of sensations and states.
Reinforcement signals in reinforcement learning are di↵erent from reward signals. The
function of a reinforcement signal is to direct the changes a learning algorithm makes inan agent’s policy, value estimates, or environment models. For a TD method, for instance,
the reinforcement signal at time t is the TD error t 1 = Rt + V (St ) V (St 1 ).1 The
1 As we mentioned in Section 6.1,

t in our notation is defined to be Rt+1 +V (St+1 )V (St ), sot15.3. The Reward Prediction Error Hypothesis

381

reinforcement signal for some algorithms could be just the reward signal, but for mostof the algorithms we consider the reinforcement signal is the reward signal adjusted byother information, such as the value estimates in TD errors.
Estimates of state values or of action values, that is, V or Q, specify what is good or
bad for the agent over the long run. They are predictions of the total reward an agent can
expect to accumulate over the future. Agents make good decisions by selecting actions
leading to states with the largest estimated state values, or by selecting actions with thelargest estimated action values.
Prediction errors measure discrepancies between expected and actual signals or sensations. Reward prediction errors (RPEs) specifically measure discrepancies between the
expected and the received reward signal, being positive when the reward signal is greaterthan expected, and negative otherwise. TD errors like (6.5) are special kinds of RPEs
that signal discrepancies between current and earlier expectations of reward over the
long-term. When neuroscientists refer to RPEs they generally (though not always) mean
TD RPEs, which we simply call TD errors throughout this chapter. Also in this chapter,a TD error is generally one that does not depend on actions, as opposed to TD errors
used in learning action-values by algorithms like Sarsa and Q-learning. This is because
the most well-known links to neuroscience are stated in terms of action-free TD errors,
but we do not mean to rule out possible similar links involving action-dependent TD
errors. (TD errors for predicting signals other than rewards are useful too, but that casewill not concern us here. See, for example, Modayil, White, and Sutton, 2014.)
One can ask many questions about links between neuroscience data and these theoreticallydefined signals. Is an observed signal more like a reward signal, a value signal, a prediction
error, a reinforcement signal, or something altogether di↵erent? And if it is an error
signal, is it an RPE, a TD error, or a simpler error like the Rescorla–Wagner error (14.3)?
And if it is a TD error, does it depend on actions like the TD error of Q-learning orSarsa? As indicated above, probing the brain to answer questions like these is extremely
difficult. But experimental evidence suggests that one neurotransmitter, specificallythe neurotransmitter dopamine, signals RPEs, and further, that the phasic activity of
dopamine-producing neurons in fact conveys TD errors (see Section 15.1 for a definition ofphasic activity). This evidence led to the reward prediction error hypothesis of dopamine
neuron activity, which we describe next.

15.3

The Reward Prediction Error Hypothesis

The reward prediction error hypothesis of dopamine neuron activity proposes that oneof the functions of the phasic activity of dopamine-producing neurons in mammals is todeliver an error between an old and a new estimate of expected future reward to target
areas throughout the brain. This hypothesis (though not in these exact words) was
first explicitly stated by Montague, Dayan, and Sejnowski (1996), who showed how theTD error concept from reinforcement learning accounts for many features of the phasic
is not available until time t + 1. The TD error available at t is actually t 1 = Rt + V (St ) V (St 1 ).
Because we are thinking of time steps as very small, or even infinitesimal, time intervals, one should not
attribute undue importance to this one-step time shift.

382

Chapter 15: Neuroscience

activity of dopamine neurons in mammals. The experiments that led to this hypothesis
were performed in the 1980s and early 1990s in the laboratory of neuroscientist Wolfram
Schultz. Section 15.5 describes these influential experiments, Section 15.6 explains how theresults of these experiments align with TD errors, and the Bibliographical and Historical
Remarks section at the end of this chapter includes a guide to the literature surrounding
the development of this influential hypothesis.
Montague et al. (1996) compared the TD errors of the TD model of classical conditioning
with the phasic activity of dopamine-producing neurons during classical conditioning
experiments. Recall from Section 14.2 that the TD model of classical conditioning isbasically the semi-gradient-descent TD( ) algorithm with linear function approximation.
Montague et al. made several assumptions to set up this comparison. First, because a
TD error can be negative but neurons cannot have a negative firing rate, they assumed
that the quantity corresponding to dopamine neuron activity is t 1 + bt , where bt is the
background firing rate of the neuron. A negative TD error corresponds to a drop in a
dopamine neuron’s firing rate below its background rate.2
A second assumption was needed about the states visited in each classical conditioning
trial and how they are represented as inputs to the learning algorithm. This is the same
issue we discussed in Section 14.2.4 for the TD model. Montague et al. chose a complete
serial compound (CSC) representation as shown in the left column of Figure 14.1, butwhere the sequence of short-duration internal signals continues until the onset of the US,
which here is the arrival of a non-zero reward signal. This representation allows the TD
error to mimic the fact that dopamine neuron activity not only predicts a future reward,
but that it is also sensitive to when after a predictive cue that reward is expected toarrive. There has to be some way to keep track of the time between sensory cues andthe arrival of reward. If a stimulus initiates a sequence of internal signals that continues
after the stimulus ends, and if there is a di↵erent signal for each time step following thestimulus, then each time step after the stimulus is represented by a distinct state. Thus,
the TD error, being state-dependent, can be sensitive to the timing of events within atrial.
In simulated trials with these assumptions about background firing rate and input
representation, TD errors of the TD model are remarkably similar to dopamine neuron
phasic activity. Previewing our description of details about these similarities in Section 15.5
below, the TD errors parallel the following features of dopamine neuron activity: (1) thephasic response of a dopamine neuron only occurs when a rewarding event is unpredicted;(2) early in learning, neutral cues that precede a reward do not cause substantial phasic
dopamine responses, but with continued learning these cues gain predictive value andcome to elicit phasic dopamine responses; (3) if an even earlier cue reliably precedes acue that has already acquired predictive value, the phasic dopamine response shifts tothe earlier cue, ceasing for the later cue; and (4) if after learning, the predicted rewarding
event is omitted, a dopamine neuron’s response decreases below its baseline level shortly
after the expected time of the rewarding event.

2 In the literature relating TD errors to the activity of dopamine neurons, theirt 1 = Rt +V (St )

V (St

1 ).

t is the same as our15.4. Dopamine

383

Although not every dopamine neuron monitored in the experiments of Schultz andcolleagues behaved in all of these ways, the striking correspondence between the activities of most of the monitored neurons and TD errors lends strong support to thereward prediction error hypothesis. There are situations, however, in which predictions
based on the hypothesis do not match what is observed in experiments. The choiceof input representation is critical to how closely TD errors match some of the detailsof dopamine neuron activity, particularly details about the timing of dopamine neuron
responses. Di↵erent ideas, some of which we discuss below, have been proposed about
input representations and other features of TD learning to make the TD errors fit the data
better, though the main parallels appear with the CSC representation that Montague etal. used. Overall, the reward prediction error hypothesis has received wide acceptanceamong neuroscientists studying reward-based learning, and it has proven to be remarkably
resilient in the face of accumulating results from neuroscience experiments.
To prepare for our description of the neuroscience experiments supporting the reward
prediction error hypothesis, and to provide some context so that the significance of the
hypothesis can be appreciated, we next present some of what is known about dopamine,
the brain structures it influences, and how it is involved in reward-based learning.

15.4

Dopamine

Dopamine is produced as a neurotransmitter by neurons whose cell bodies lie mainly
in two clusters of neurons in the midbrain of mammals: the substantia nigra pars
compacta (SNpc) and the ventral tegmental area (VTA). Dopamine plays essential roles
in many processes in the mammalian brain. Prominent among these are motivation,
learning, action-selection, most forms of addiction, and the disorders schizophrenia andParkinson’s disease. Dopamine is called a neuromodulator because it performs many
functions other than direct fast excitation or inhibition of targeted neurons. Althoughmuch remains unknown about dopamine’s functions and details of its cellular e↵ects, it isclear that it is fundamental to reward processing in the mammalian brain. Dopamine
is not the only neuromodulator involved in reward processing, and its role in aversive
situations—punishment—remains controversial. Dopamine also can function di↵erently in
non-mammals. But no one doubts that dopamine is essential for reward-related processes
in mammals, including humans.
An early, traditional view is that dopamine neurons broadcast a reward signal tomultiple brain regions implicated in learning and motivation. This view followed from afamous 1954 paper by James Olds and Peter Milner that described the e↵ects of electricalstimulation on certain areas of a rat’s brain. They found that electrical stimulation toparticular regions acted as a very powerful reward in controlling the rat’s behavior: “...thecontrol exercised over the animal’s behavior by means of this reward is extreme, possibly
exceeding that exercised by any other reward previously used in animal experimentation”(Olds and Milner, 1954). Later research revealed that the sites at which stimulation
was most e↵ective in producing this rewarding e↵ect excited dopamine pathways, either
directly or indirectly, that ordinarily are excited by natural rewarding stimuli. E↵ects
similar to these were also observed with human subjects. These observations strongly
suggested that dopamine neuron activity signals reward.

384

Chapter 15: Neuroscience

But if the reward prediction error hypothesis is correct—even if it accounts for only
some features of a dopamine neuron’s activity—this traditional view of dopamine neuron
activity is not entirely correct: phasic responses of dopamine neurons signal reward
prediction errors, not reward itself. In reinforcement learning’s terms, a dopamine
neuron’s phasic response at a time t corresponds to t 1 = Rt + V (St ) V (St 1 ), not
to Rt .
Reinforcement learning theory and algorithms help reconcile the reward-predictionerror view with the conventional notion that dopamine signals reward. In many of thealgorithms we discuss in this book, functions as a reinforcement signal, meaning that itis the main driver of learning. For example, is the critical factor in the TD model ofclassical conditioning, and is the reinforcement signal for learning both a value function
and a policy in an actor–critic architecture (Sections 13.5 and 15.7). Action-dependent
forms of are reinforcement signals for Q-learning and Sarsa. The reward signal Rt isa crucial component of t 1 , but it is not the complete determinant of its reinforcing
e↵ect in these algorithms. The additional term V (St ) V (St 1 ) is the higher-orderreinforcement part of t 1 , and even if reward occurs (Rt 6= 0), the TD error can be silent
if the reward is fully predicted (which is fully explained in Section 15.6 below).
A closer look at Olds’ and Milner’s 1954 paper, in fact, reveals that it is mainly
about the reinforcing e↵ect of electrical stimulation in an instrumental conditioning task.
Electrical stimulation not only energized the rats’ behavior—through dopamine’s e↵ect on
motivation—it also led to the rats quickly learning to stimulate themselves by pressing alever, which they would do frequently for long periods of time. The activity of dopamine
neurons triggered by electrical stimulation reinforced the rats’ lever pressing.
More recent experiments using optogenetic methods clinch the role of phasic responsesof dopamine neurons as reinforcement signals. These methods allow neuroscientists toprecisely control the activity of selected neuron types at a millisecond timescale in awake
behaving animals. Optogenetic methods introduce light-sensitive proteins into selected
neuron types so that these neurons can be activated or silenced by means of flashes oflaser light. The first experiment using optogenetic methods to study dopamine neurons
showed that optogenetic stimulation producing phasic activation of dopamine neurons
in mice was enough to condition the mice to prefer the side of a chamber where theyreceived this stimulation as compared to the chamber’s other side where they received
no, or lower-frequency, stimulation (Tsai et al. 2009). In another example, Steinberg
et al. (2013) used optogenetic activation of dopamine neurons to create artificial burstsof dopamine neuron activity in rats at the times when rewarding stimuli were expected
but omitted—times when dopamine neuron activity normally pauses. With these pauses
replaced by artificial bursts, responding was sustained when it would ordinarily decrease
due to lack of reinforcement (in extinction trials), and learning was enabled when it would
ordinarily be blocked due to the reward being already predicted (the blocking paradigm;
Section 14.2.1).
Additional evidence for the reinforcing function of dopamine comes from optogenetic
experiments with fruit flies, except in these animals dopamine’s e↵ect is the opposite ofits e↵ect in mammals: optically triggered bursts of dopamine neuron activity act justlike electric foot shock in reinforcing avoidance behavior, at least for the population

15.4. Dopamine385

of dopamine neurons activated (Claridge-Chang et al. 2009). Although none of these
optogenetic experiments showed that phasic dopamine neuron activity is specificallylike a TD error, they convincingly demonstrated that phasic dopamine neuron activity
acts just like acts (or perhaps like minus acts in fruit flies) as the reinforcement
signal in algorithms for both prediction (classical conditioning) and control (instrumental
conditioning).
Dopamine neurons are particularly well suited
to broadcasting a reinforcement signal to many
areas of the brain. These neurons have hugeaxonal arbors, each releasing dopamine at 100 to
1,000 times more synaptic sites than reached by
the axons of typical neurons. Shown to the rightis the axonal arbor of a single dopamine neuron
whose cell body is in the SNpc of a rat’s brain.
Each axon of a SNpc or VTA dopamine neuron
makes roughly 500,000 synaptic contacts on thedendrites of neurons in targeted brain areas.
If dopamine neurons broadcast a reinforcement signal like reinforcement learning’s , thenbecause this is a scalar signal, i.e., a single number, all dopamine neurons in both the SNpc
and VTA would be expected to activate moreor-less identically so that they would act in nearsynchrony to send the same signal to all of the Axonal arbor of a single neuron producing
sites their axons target. Although it has been dopamine as a neurotransmitter. Thesea common belief that dopamine neurons do act axons make synaptic contacts with a huge
together like this, modern evidence is pointing number of dendrites of neurons in targetedto the more complicated picture that di↵erent brain areas.
subpopulations of dopamine neurons respond to Adapted from The Journal of Neuroscience,
Matsuda, Furuta, Nakamura, Hioki, Fujiyama,
input di↵erently depending on the structures to Arai, and Kaneko, volume 29, 2009, page 451.
which they send their signals and the di↵erent
ways these signals act on their target structures. Dopamine has functions other thansignaling RPEs, and even for dopamine neurons that do signal RPEs, it can make sense
to send di↵erent RPEs to di↵erent structures depending on the roles these structures
play in producing reinforced behavior. This is beyond what we treat in any detail in thisbook, but vector-valued RPE signals make sense from the perspective of reinforcement
learning when decisions can be decomposed into separate sub-decisions, or more generally,
as a way to address the structural version of the credit assignment problem: How do
you distribute credit for success (or blame for failure) of a decision among the many
component structures that could have been involved in producing it? We say a bit more
about this in Section 15.10 below.
The axons of most dopamine neurons make synaptic contact with neurons in the frontal
cortex and the basal ganglia, areas of the brain involved in voluntary movement, decision
making, learning, and cognitive functions such as planning. Because most ideas relating

386

Chapter 15: Neuroscience

dopamine to reinforcement learning focus on the basal ganglia, and the connections fromdopamine neurons are particularly dense there, we focus on the basal ganglia here. The
basal ganglia are a collection of neuron groups, or nuclei, lying at the base of the forebrain.
The main input structure of the basal ganglia is called the striatum. Essentially all of thecerebral cortex, among other structures, provides input to the striatum. The activity ofcortical neurons conveys a wealth of information about sensory input, internal states, andmotor activity. The axons of cortical neurons make synaptic contacts on the dendrites ofthe main input/output neurons of the striatum, called medium spiny neurons. Output
from the striatum loops back via other basal ganglia nuclei and the thalamus to frontal
areas of cortex, and to motor areas, making it possible for the striatum to influence
movement, abstract decision processes, and reward processing. Two main subdivisionsof the striatum are important for reinforcement learning: the dorsal striatum, primarily
implicated in influencing action selection, and the ventral striatum, thought to be criticalfor di↵erent aspects of reward processing, including the assignment of a↵ective value tosensations.
The dendrites of medium spiny neurons are covered with spines on whose tips theaxons of neurons in the cortex make synaptic contact. Also making synaptic contact withthese spines—in this case contacting the spine stems—are axons of dopamine neurons(Figure 15.1). This arrangement brings together presynaptic activity of cortical neurons,

Figure 15.1: Spine of a striatal neuron showing input from both cortical and dopamine neurons.
Axons of cortical neurons influence striatal neurons via corticostriatal synapses releasing theneurotransmitter glutamate at the tips of spines covering the dendrites of striatal neurons.
An axon of a VTA or SNpc dopamine neuron is shown passing by the spine (from the lower
right). “Dopamine varicosities” on this axon release dopamine at or near the spine stem, in an
arrangement that brings together presynaptic input from cortex, postsynaptic activity of thestriatal neuron, and dopamine, making it possible that several types of learning rules govern theplasticity of corticostriatal synapses. Each axon of a dopamine neuron makes synaptic contact
with the stems of roughly 500,000 spines. Some of the complexity omitted from our discussion
is shown here by other neurotransmitter pathways and multiple receptor types, such as D1 an
D2 dopamine receptors by which dopamine can produce di↵erent e↵ects at spines and otherpostsynaptic sites. From Journal of Neurophysiology, W. Schultz, vol. 80, 1998, page 10.

15.5. Experimental Support for the Reward Prediction Error Hypothesis

387

postsynaptic activity of medium spiny neurons, and input from dopamine neurons. Whatactually occurs at these spines is complex and not completely understood. Figure 15.1
hints at the complexity by showing two types of receptors for dopamine, receptors forglutamate—the neurotransmitter of the cortical inputs—and multiple ways that thevarious signals can interact. But evidence is mounting that changes in the efficacies ofthe synapses on the pathway from the cortex to the striatum, which neuroscientists callcorticostriatal synapses, depend critically on appropriately-timed dopamine signals.

15.5

Experimental Support
for the Reward Prediction Error Hypothesis

Dopamine neurons respond with bursts of activity to intense, novel, or unexpected visualand auditory stimuli that trigger eye and body movements, but very little of their activity isrelated to the movements themselves. This is surprising because degeneration of dopamine
neurons is a cause of Parkinson’s disease, whose symptoms include motor disorders,
particularly deficits in self-initiated movement. Motivated by the weak relationshipbetween dopamine neuron activity and stimulus-triggered eye and body movements,
Romo and Schultz (1990) and Schultz and Romo (1990) took the first steps toward thereward prediction error hypothesis by recording the activity of dopamine neurons andmuscle activity while monkeys moved their arms.
They trained two monkeys to reach from a resting hand position into a bin containing
a bit of apple, a piece of cookie, or a raisin, when the monkey saw and heard the bin’s
door open. The monkey could then grab and bring the food to its mouth. After a monkey
became good at this, it was trained on two additional tasks. The purpose of the firsttask was to see what dopamine neurons do when movements are self-initiated. The bin
was left open but covered from above so that the monkey could not see inside but could
reach in from below. No triggering stimuli were presented, and after the monkey reached
for and ate the food morsel, the experimenter usually (though not always), silently andunseen by the monkey, replaced food in the bin by sticking it onto a rigid wire. Here too,
the activity of the dopamine neurons Romo and Schultz monitored was not related to themonkey’s movements, but a large percentage of these neurons produced phasic responses
whenever the monkey first touched a food morsel. These neurons did not respond whenthe monkey touched just the wire or explored the bin when no food was there. This wasgood evidence that the neurons were responding to the food and not to other aspects ofthe task.
The purpose of Romo and Schultz’s second task was to see what happens whenmovements are triggered by stimuli. This task used a di↵erent bin with a movable cover.
The sight and sound of the bin opening triggered reaching movements to the bin. In thiscase, Romo and Schultz found that after some period of training, the dopamine neurons
no longer responded to the touch of the food but instead responded to the sight and soundof the opening cover of the food bin. The phasic responses of these neurons had shiftedfrom the reward itself to stimuli predicting the availability of the reward. In a followup
study, Romo and Schultz found that most of the dopamine neurons whose activity they

388

Chapter 15: Neuroscience

monitored did not respond to the sight and sound of the bin opening outside the context
of the behavioral task. These observations suggested that the dopamine neurons were
responding neither to the initiation of a movement nor to the sensory properties of thestimuli, but were rather signaling an expectation of reward.
Schultz’s group conducted many additional studies involving both SNpc and VTA
dopamine neurons. A particular series of experiments was influential in suggesting thatthe phasic responses of dopamine neurons correspond to TD errors and not to simpler
errors like those in the Rescorla–Wagner model (14.3). In the first of these experiments(Ljungberg, Apicella, and Schultz, 1992), monkeys were trained to depress a lever after
a light was illuminated as a ‘trigger cue’ to obtain a drop of apple juice. As Romoand Schultz had observed earlier, many dopamine neurons initially responded to thereward—the drop of juice (Figure 15.2, top panel). But many of these neurons lost thatreward response as training continued and developed responses instead to the illuminationof the light that predicted the reward (Figure 15.2, middle panel). With continued
training, lever pressing became faster while the number of dopamine neurons responding
to the trigger cue decreased.
Following this study, the same monkeys were trained on a new task (Schultz, Apicella,and Ljungberg, 1993). Here the monkeys faced two levers, each with a light above it.
Illuminating one of these lights was an ‘instruction cue’ indicating which of the two levers

Figure 15.2: The response of dopamine neurons shifts from initial responses to primary reward
to earlier predictive stimuli. These are plots of the number of action potentials producedby monitored dopamine neurons within small time intervals, averaged over all the monitored
dopamine neurons (ranging from 23 to 44 neurons for these data). Top: dopamine neurons areactivated by the unpredicted delivery of drop of apple juice. Middle: with learning, dopamine
neurons developed responses to the reward-predicting trigger cue and lost responsiveness to thedelivery of reward. Bottom: with the addition of an instruction cue preceding the trigger cue by
1 second, dopamine neurons shifted their responses from the trigger cue to the earlier instruction
cue. From Schultz et al. (1995), MIT Press.

15.5. Experimental Support for the Reward Prediction Error Hypothesis

would produce a drop of apple juice.
In this task, the instruction cue preceded the trigger cue of the previous task by a fixed interval of 1 second. The monkeys learned to withhold reaching until seeing the trigger cue, and dopamine neuron activity increased, but now the responsesof the monitored dopamine neurons
occurred almost exclusively to theearlier instruction cue and not to
the trigger cue (Figure 15.2, bottom panel). Here again the number of dopamine neurons responding
to the instruction cue was much reduced when the task was well learned.
During learning across these tasks,
dopamine neuron activity shiftedfrom initially responding to the reward to responding to the earlier
predictive stimuli, first progressing
to the trigger stimulus then to thestill earlier instruction cue. As responding moved earlier in time itdisappeared from the later stimuli.
This shifting of responses to earlier
reward predictors, while losing responses to later predictors is a hallmark of TD learning (see, for example, Figure 14.2).
The task just described revealed
another property of dopamine neuron activity shared with TD learning. The monkeys sometimes pressed
the wrong key, that is, the key otherthan the instructed one, and consequently received no reward. In thesetrials, many of the dopamine neurons showed a sharp decrease in theirfiring rates below baseline shortly after the reward’s usual time of delivery, and this happened without theavailability of any external cue tomark the usual time of reward delivery (Figure 15.3). Somehow the389

Figure 15.3: The response of dopamine neurons drops
below baseline shortly after the time when an expected
reward fails to occur. Top: dopamine neurons areactivated by the unpredicted delivery of a drop ofapple juice. Middle: dopamine neurons respond to aconditioned stimulus (CS) that predicts reward anddo not respond to the reward itself. Bottom: when
the reward predicted by the CS fails to occur, theactivity of dopamine neurons drops below baseline
shortly after the time the reward is expected to occur.
At the top of each of these panels is shown the average
number of action potentials produced by monitored
dopamine neurons within small time intervals aroundthe indicated times. The raster plots below show theactivity patterns of the individual dopamine neurons
that were monitored; each dot represents an action
potential. From Schultz, Dayan, and Montague, A
Neural Substrate of Prediction and Reward, Science,
vol. 275, issue 5306, pages 1593-1598, March 14, 1997.
Reprinted with permission from AAAS.

390

Chapter 15: Neurosciencemonkeys were internally keeping track of the timing of the reward. (Response timing is
one area where the simplest version of TD learning needs to be modified to account forsome of the details of the timing of dopamine neuron responses. We consider this issue inthe following section.)
The observations from the studies described above led Schultz and his group to conclude
that dopamine neurons respond to unpredicted rewards, to the earliest predictors of
reward, and that dopamine neuron activity decreases below baseline if a reward, or apredictor of reward, does not occur at its expected time. Researchers familiar withreinforcement learning were quick to recognize that these results are strikingly similar
to how the TD error behaves as the reinforcement signal in a TD algorithm. The next
section explores this similarity by working through a specific example in detail.

15.6

TD Error/Dopamine CorrespondenceThis section explains the correspondence between the TD error and the phasic responsesof dopamine neurons observed in the experiments just described. We examine howchanges over the course of learning in a task something like the one described above
where a monkey first sees an instruction cue and then a fixed time later has to respondcorrectly to a trigger cue in order to obtain reward. We use a simple idealized version ofthis task, but we go into a lot more detail than is usual because we want to emphasize
the theoretical basis of the parallel between TD errors and dopamine neuron activity.
The first simplifying assumption is that the agent has already learned the actions
required to obtain reward. Then its task is just to learn accurate predictions of future
reward for the sequence of states it experiences. This is then a prediction task, ormore technically, a policy-evaluation task: learning the value function for a fixed policy(Sections 4.1 and 6.1). The value function to be learned assigns to each state a value thatpredicts the return that will follow that state if the agent selects actions according to thegiven policy, where the return is the (possibly discounted) sum of all the future rewards.
This is unrealistic as a model of the monkey’s situation because the monkey would likely
learn these predictions at the same time that it is learning to act correctly (as would areinforcement learning algorithm that learns policies as well as value functions, such as
an actor–critic algorithm), but this scenario is simpler to describe than one in which a
policy and a value function are learned simultaneously.
Now imagine that the agent’s experience divides into multiple trials, in each of which
the same sequence of states repeats, with a distinct state occurring on each time stepduring the trial. Further imagine that the return being predicted is limited to the return
over a trial, which makes a trial analogous to a reinforcement learning episode as we havedefined it. In reality, of course, the returns being predicted are not confined to single
trials, and the time interval between trials is an important factor in determining what an
animal learns. This is true for TD learning as well, but here we assume that returns do
not accumulate over multiple trials. Given this, then, a trial in experiments like those
conducted by Schultz and colleagues is equivalent to an episode of reinforcement learning.
(Though in this discussion, we will use the term trial instead of episode to relate betterto the experiments.)

15.6. TD Error/Dopamine Correspondence391

As usual, we also need to make an assumption about how states are represented as
inputs to the learning algorithm, an assumption that influences how closely the TD error
corresponds to dopamine neuron activity. We discuss this issue later, but for now weassume the same CSC representation used by Montague et al. (1996) in which there is aseparate internal stimulus for each state visited at each time step in a trial. This reduces
the process to the tabular case covered in the first part of this book. Finally, we assume
that the agent uses TD(0) to learn a value function, V , stored in a lookup table initialized
to be zero for all the states. We also assume that this is a deterministic task and thatthe discount factor, , is very nearly one so that we can ignore it.
Figure 15.4 shows the time courses of R, V , and at several stages of learning in thispolicy-evaluation task. The time axes represent the time interval over which a sequenceof states is visited in a trial (where for clarity we omit showing individual states). The
reward signal is zero throughout each trial except when the agent reaches the rewarding
state, shown near the right end of the time line, when the reward signal becomes somepositive number, say R? . The goal of TD learning is to predict the return for eachstate visited in a trial, which in this undiscounted case and given our assumption thatpredictions are confined to individual trials, is simply R? for each state.

Rt= Rt + Vt
t 1
R
R
R

early inlearningVlearning
complete

V

Vt 1

R?

R over this interval
regular predictors of R

R omitted
R
R

Figure 15.4: The behavior of the TD error during TD learning is consistent with features ofthe phasic activation of dopamine neurons. (Here is the TD error available at time t, i.e., t 1 ).
Top: a sequence of states, shown as an interval of regular predictors, is followed by a non-zeroreward R? . Early in learning: the initial value function, V , and initial , which at first is equal
to R? . Learning complete: the value function accurately predicts future reward, is positive at
the earliest predictive state, and = 0 at the time of the non-zero reward. R? omitted : at thetime the predicted reward is omitted, becomes negative. See text for a complete explanationof why this happens.

392

Chapter 15: Neuroscience

Preceding the rewarding state is a sequence of reward-predicting states, with theearliest reward-predicting state shown near the left end of the time line. This is likethe state near the start of a trial, for example like the state marked by the instruction
cue in a trial of the monkey experiment of Schultz et al. (1993) described above. It is
the first state in a trial that reliably predicts that trial’s reward. (Of course, in reality
states visited on preceding trials are even earlier reward-predicting states, but because
we are confining predictions to individual trials, these do not qualify as predictors of thistrial’s reward. Below we give a more satisfactory, though more abstract, description of an
earliest reward-predicting state.) The latest reward-predicting state in a trial is the state
immediately preceding the trial’s rewarding state. This is the state near the far right endof the time line in Figure 15.4. Note that the rewarding state of a trial does not predictthe return for that trial: the value of this state would come to predict the return over all
the following trials, which here we are assuming to be zero in this episodic formulation.
Figure 15.4 shows the first-trial time courses of V and as the graphs labeled ‘early inlearning.’ Because the reward signal is zero throughout the trial except when the rewarding
state is reached, and all the V -values are zero, the TD error is also zero until it becomesR? at the rewarding state. This follows because t 1 = Rt + Vt Vt 1 = Rt + 0 0 = Rt ,which is zero until it equals R? when the reward occurs. Here Vt and Vt 1 are respectively
the estimated values of the states visited at times t and t 1 in a trial. The TD error at
this stage of learning is analogous to a dopamine neuron responding to an unpredicted
reward (e.g., a drop of apple juice) at the start of training.
Throughout this first trial and all successive trials, TD(0) updates occur at eachstate transition as described in Chapter 6. This successively increases the values of the
reward-predicting states, with the increases spreading backwards from the rewarding
state, until the values converge to the correct return predictions. In this case (because
we are assuming no discounting) the correct predictions are equal to R? for all the
reward-predicting states. This can be seen in Figure 15.4 as the graph of V labeled‘learning complete’ where the values of all the states from the earliest to the latest
reward-predicting states all equal R? . The values of the states preceding the earliest
reward-predicting state remain low (which Figure 15.4 shows as zero) because they are
not reliable predictors of reward.
When learning is complete, that is, when V attains its correct values, the TD errors
associated with transitions from any reward-predicting state are zero because the predictions are now accurate. This is because for a transition from a reward-predicting state toanother reward-predicting state, we have t 1 = Rt + Vt Vt 1 = 0 + R? R? = 0, andfor the transition from the latest reward-predicting state to the rewarding state, we haveVt 1 = R? + 0 R? = 0. On the other hand, the TD error on a transitiont 1 = Rt + Vt
from any state to the earliest reward-predicting state is positive because of the mismatch
between this state’s low value and the larger value of the following reward-predicting
state. Indeed, if the value of a state preceding the earliest reward-predicting state werezero, then after the transition to the earliest reward-predicting state, we would havethat t 1 = Rt + Vt Vt 1 = 0 + R? 0 = R? . The ‘learning complete’ graph of in
Figure 15.4 shows this positive value at the earliest reward-predicting state, and zeros
everywhere else.

15.6. TD Error/Dopamine Correspondence

393

The positive TD error upon transitioning to the earliest reward-predicting state isanalogous to the persistence of dopamine responses to the earliest stimuli predicting
reward. By the same token, when learning is complete, a transition from the latest
reward-predicting state to the rewarding state produces a zero TD error because thelatest reward-predicting state’s value, being correct, cancels the reward. This parallels theobservation that fewer dopamine neurons generate a phasic response to a fully predicted
reward than to an unpredicted reward.
After learning, if the reward is suddenly omitted, the TD error goes negative at theusual time of reward because the value of the latest reward-predicting state is then too
high: t 1 = Rt + Vt Vt 1 = 0 + 0 R? = R? , as shown at the right end of the ‘Romitted’ graph of in Figure 15.4. This is like dopamine neuron activity decreasing
below baseline at the time an expected reward is omitted as seen in the experiment of
Schultz et al. (1993) described above and shown in Figure 15.3.
The idea of an earliest reward-predicting state deserves more attention. In the scenario
described above, because experience is divided into trials, and we assumed that predictions
are confined to individual trials, the earliest reward-predicting state is always the firststate of a trial. Clearly this is artificial. A more general way to think of an earliest
reward-predicting state is that it is an unpredicted predictor of reward, and there canbe many such states. In an animal’s life, many di↵erent states may precede an earliest
reward-predicting state. However, because these states are more often followed by otherstates that do not predict reward, their reward-predicting powers, that is, their values,
remain low. A TD algorithm, if operating throughout the animal’s life, would update thevalues of these states too, but the updates would not consistently accumulate because, byassumption, none of these states reliably precedes an earliest reward-predicting state. If
any of them did, they would be reward-predicting states as well. This might explain why
with overtraining, dopamine responses decrease to even the earliest reward-predicting
stimulus in a trial. With overtraining one would expect that even a formerly-unpredictedpredictor state would become predicted by stimuli associated with earlier states: theanimal’s interaction with its environment both inside and outside of an experimental
task would become commonplace. Upon breaking this routine with the introduction of a
new task, however, one would see TD errors reappear, as indeed is observed in dopamine
neuron activity.
The example described above explains why the TD error shares key features withthe phasic activity of dopamine neurons when the animal is learning in a task similar
to the idealized task of our example. But not every property of the phasic activity of
dopamine neurons coincides so neatly with properties of . One of the most troubling
discrepancies involves what happens when a reward occurs earlier than expected. We
have seen that the omission of an expected reward produces a negative prediction error
at the reward’s expected time, which corresponds to the activity of dopamine neurons
decreasing below baseline when this happens. If the reward arrives later than expected,
it is then an unexpected reward and generates a positive prediction error. This happens
with both TD errors and dopamine neuron responses. But when reward arrives earlier
than expected, dopamine neurons do not do what the TD error does—at least with the
CSC representation used by Montague et al. (1996) and by us in our example. Dopamine

394

Chapter 15: Neuroscience

neurons do respond to the early reward, which is consistent with a positive TD error
because the reward is not predicted to occur then. However, at the later time when the
reward is expected but omitted, the TD error is negative whereas, in contrast to thisprediction, dopamine neuron activity does not drop below baseline in the way the TD
model predicts (Hollerman and Schultz, 1998). Something more complicated is going onin the animal’s brain than simply TD learning with a CSC representation.
Some of the mismatches between the TD error and dopamine neuron activity canbe addressed by selecting suitable parameter values for the TD algorithm and by using
stimulus representations other than the CSC representation. For instance, to address
the early-reward mismatch just described, Suri and Schultz (1999) proposed a CSC
representation in which the sequences of internal signals initiated by earlier stimuli
are cancelled by the occurrence of a reward. Another proposal by Daw, Courville,
and Touretzky (2006) is that the brain’s TD system uses representations produced bystatistical modeling carried out in sensory cortex rather than simpler representations
based on raw sensory input. Ludvig, Sutton, and Kehoe (2008) found that TD learningwith a microstimulus (MS) representation (Figure 14.1) fits the activity of dopamine
neurons in the early-reward and other situations better than when a CSC representation
is used. Pan, Schmidt, Wickens, and Hyland (2005) found that even with the CSC
representation, prolonged eligibility traces improve the fit of the TD error to some aspectsof dopamine neuron activity. In general, many fine details of TD-error behavior depend
on subtle interactions between eligibility traces, discounting, and stimulus representations.
Findings like these elaborate and refine the reward prediction error hypothesis without
refuting its core claim that the phasic activity of dopamine neurons is well characterized
as signaling TD errors.
On the other hand, there are other discrepancies between the TD theory and experimental data that are not so easily accommodated by selecting parameter values andstimulus representations (we mention some of these discrepancies in the Bibliographicaland Historical Remarks section at the end of this chapter), and more mismatches arelikely to be discovered as neuroscientists conduct ever more refined experiments. But the
reward prediction error hypothesis has been functioning very e↵ectively as a catalyst for
improving our understanding of how the brain’s reward system works. Intricate experiments have been designed to validate or refute predictions derived from the hypothesis,
and experimental results have, in turn, led to refinement and elaboration of the TD
error/dopamine hypothesis.
A remarkable aspect of these developments is that the reinforcement learning algorithms and theory that connect so well with properties of the dopamine system weredeveloped from a computational perspective in total absence of any knowledge about therelevant properties of dopamine neurons—remember, TD learning and its connections
to optimal control and dynamic programming were developed many years before any ofthe experiments were conducted that revealed the TD-like nature of dopamine neuron
activity. This unplanned correspondence, despite not being perfect, suggests that the TD
error/dopamine parallel captures something significant about brain reward processes.
In addition to accounting for many features of the phasic activity of dopamine neurons,
the reward prediction error hypothesis links neuroscience to other aspects of reinforcement15.7. Neural Actor–Critic

395

learning, in particular, to learning algorithms that use TD errors as reinforcement signals.
Neuroscience is still far from reaching complete understanding of the circuits, molecularmechanisms, and functions of the phasic activity of dopamine neurons, but evidence
supporting the reward prediction error hypothesis, along with evidence that phasic
dopamine responses are reinforcement signals for learning, suggest that the brain might
implement something like an actor–critic algorithm in which TD errors play critical roles.
Other reinforcement learning algorithms are plausible candidates too, but actor–critic
algorithms fit the anatomy and physiology of the mammalian brain particularly well, as
we describe in the following two sections.

15.7

Neural Actor–Critic

Actor–critic algorithms learn both policies and value functions. The ‘actor’ is thecomponent that learns policies, and the ‘critic’ is the component that learns about
whatever policy is currently being followed by the actor in order to ‘criticize’ the actor’s
action choices. The critic uses a TD algorithm to learn the state-value function for theactor’s current policy. The value function allows the critic to critique the actor’s action
choices by sending TD errors, , to the actor. A positive means that the action was‘good’ because it led to a state with a better-than-expected value; a negative meansthat the action was ‘bad’ because it led to a state with a worse-than-expected value.
Based on these critiques, the actor continually updates its policy.
Two distinctive features of actor–critic algorithms are responsible for thinking that thebrain might implement an algorithm like this. First, the two components of an actor–critic
algorithm—the actor and the critic—suggest that two parts of the striatum—the dorsal
and ventral subdivisions (Section 15.4), both critical for reward-based learning—mayfunction respectively something like an actor and a critic. A second property of actor–
critic algorithms that suggests a brain implementation is that the TD error has the dualrole of being the reinforcement signal for both the actor and the critic, though it has adi↵erent influence on learning in each of these components. This fits well with several
properties of the neural circuitry: axons of dopamine neurons target both the dorsal
and ventral subdivisions of the striatum; dopamine appears to be critical for modulating
synaptic plasticity in both structures; and how a neuromodulator such as dopamine
acts on a target structure depends on properties of the target structure and not just on
properties of the neuromodulator.
Section 13.5 presents actor–critic algorithms as policy gradient methods, but the actor–
critic algorithm of Barto, Sutton, and Anderson (1983) was simpler and was presented as
an artificial neural network (ANN). Here we describe an ANN implementation something
like that of Barto et al., and we follow Takahashi, Schoenbaum, and Niv (2008) in giving
a schematic proposal for how this ANN might be implemented by real neural networks inthe brain. We postpone discussion of the actor and critic learning rules until Section 15.8,
where we present them as special cases of the policy-gradient formulation and discuss
what they suggest about how dopamine might modulate synaptic plasticity.

396

Chapter 15: Neuroscience

Figure 15.5a shows an implementation of an actor–critic algorithm as an ANN withcomponent networks implementing the actor and the critic. The critic consists of a single
neuron-like unit, V , whose output activity represents state values, and a component
shown as the diamond labeled TD that computes TD errors by combining V ’s output
with reward signals and with previous state values (as suggested by the loop from the
TD diamond to itself). The actor network has a single layer of k actor units labeled Ai ,i = 1, . . . , k. The output of each actor unit is a component of a k-dimensional action
vector. An alternative is that there are k separate actions, one commanded by each actor
unit, that compete with one another to be executed, but here we will think of the entire
A-vector as an action.

(a)

(b)

Environment

Environment

TD error δ

A1

x1
x22

A2
A3

.
.
.

.
.
.

xn

Ventralstriatum

.
.
.

VTA
SNc

Sn

Actions

xnn

TD

Critic

S2

Dopamine

S1
S2

.
.
.

.
.
.

Dorsal striatum

V

.
.
.

S1

Cortex (multiple areas)

CriticStates/Stimuli

x11
x2

RewardActions

States/Stimuli

Reward

Sn

Actor

Ak

Actor

Figure 15.5: Actor–critic ANN and a hypothetical neural implementation. a) Actor–critic
algorithm as an ANN. The actor adjusts a policy based on the TD error it receives from thecritic; the critic adjusts state-value parameters using the same . The critic produces a TD error
from the reward signal, R, and the current change in its estimate of state values. The actor does
not have direct access to the reward signal, and the critic does not have direct access to theaction. b) Hypothetical neural implementation of an actor–critic algorithm. The actor and thevalue-learning part of the critic are respectively placed in the dorsal and ventral subdivisionsof the striatum. The TD error is transmitted by dopamine neurons located in the VTA andSNpc to modulate changes in synaptic efficacies of input from cortical areas to the ventral anddorsal striatum. Adapted from Frontiers in Neuroscience, vol. 2(1), 2008, Y. Takahashi, G.
Schoenbaum, and Y. Niv, Silencing the critics: Understanding the e↵ects of cocaine sensitization
on dorsolateral and ventral striatum in the context of an Actor/Critic model.

15.7. Neural Actor–Critic

397

Both the critic and actor networks receive input consisting of multiple features representing the state of the agent’s environment. (Recall from Chapter 1 that the environmentof a reinforcement learning agent includes components both inside and outside of the‘organism’ containing the agent.) The figure shows these features as the circles labeled
x1 , x2 , . . . , xn , shown twice just to keep the figure simple. A weight representing theefficacy of a synapse is associated with each connection from each feature xi to thecritic unit, V , and to each of the action units, Ai . The weights in the critic network
parameterize the value function, and the weights in the actor network parameterize thepolicy. The networks learn as these weights change according to the critic and actor
learning rules that we describe in the following section.
The TD error produced by circuitry in the critic is the reinforcement signal for changing
the weights in both the critic and the actor networks. This is shown in Figure 15.5a by
the line labeled ‘TD error ’ extending across all of the connections in the critic andactor networks. This aspect of the network implementation, together with the reward
prediction error hypothesis and the fact that the activity of dopamine neurons is so
widely distributed by the extensive axonal arbors of these neurons, suggests that an
actor–critic network something like this may not be too farfetched as a hypothesis about
how reward-related learning might happen in the brain.
Figure 15.5b suggests—very schematically—how the ANN on the figure’s left might
map onto structures in the brain according to the hypothesis of Takahashi et al. (2008).
The hypothesis puts the actor and the value-learning part of the critic respectively in thedorsal and ventral subdivisions of the striatum, the input structure of the basal ganglia.
Recall from Section 15.4 that the dorsal striatum is primarily implicated in influencing
action selection, and the ventral striatum is thought to be critical for di↵erent aspects ofreward processing, including the assignment of a↵ective value to sensations. The cerebral
cortex, along with other structures, sends input to the striatum conveying information
about stimuli, internal states, and motor activity.
In this hypothetical actor–critic brain implementation, the ventral striatum sends value
information to the VTA and SNpc, where dopamine neurons in these nuclei combine itwith information about reward to generate activity corresponding to TD errors (though
exactly how dopaminergic neurons calculate these errors is not yet understood). The ‘TD
error ’ line in Figure 15.5a becomes the line labeled ‘Dopamine’ in Figure 15.5b, which
represents the widely branching axons of dopamine neurons whose cell bodies are in the
VTA and SNpc. Referring back to Figure 15.1, these axons make synaptic contact withthe spines on the dendrites of medium spiny neurons, the main input/output neurons ofboth the dorsal and ventral divisions of the striatum. Axons of the cortical neurons that
send input to the striatum make synaptic contact on the tips of these spines. According
to the hypothesis, it is at these spines where changes in the efficacies of the synapses
from cortical regions to the striatum are governed by learning rules that critically depend
on a reinforcement signal supplied by dopamine.
An important implication of the hypothesis illustrated in Figure 15.5b is that the
dopamine signal is not the ‘master’ reward signal like the scalar Rt of reinforcement
learning. In fact, the hypothesis implies that one should not necessarily be able toprobe the brain and record any signal like Rt in the activity of any single neuron.

398

Chapter 15: Neuroscience

Many interconnected neural systems generate reward-related information, with di↵erent
structures being recruited depending on di↵erent types of rewards. Dopamine neurons
receive information from many di↵erent brain areas, so the input to the SNpc and
VTA labeled ‘Reward’ in Figure 15.5b should be thought of as vector of reward-related
information arriving to neurons in these nuclei along multiple input channels. What thetheoretical scalar reward signal Rt might correspond to, then, is the net contribution of
all reward-related information to dopamine neuron activity. It is the result of a patternof activity across many neurons in di↵erent areas of the brain.
Although the actor–critic neural implementation illustrated in Figure 15.5b may becorrect on some counts, it clearly needs to be refined, extended, and modified to qualify
as a full-fledged model of the function of the phasic activity of dopamine neurons. The
Historical and Bibliographic Remarks section at the end of this chapter cites publications
that discuss in more detail both empirical support for this hypothesis and places where it
falls short. We now look in detail at what the actor and critic learning algorithms suggest
about the rules governing changes in synaptic efficacies of corticostriatal synapses.

15.8

Actor and Critic Learning Rules

If the brain does implement something like the actor–critic algorithm—and assuming
populations of dopamine neurons broadcast a common reinforcement signal to the corticostriatal synapses of both the dorsal and ventral striatum as illustrated in Figure 15.5b(which is likely an oversimplification as we mentioned above)—then this reinforcement
signal a↵ects the synapses of these two structures in di↵erent ways. The learning rules forthe critic and the actor use the same reinforcement signal, the TD error , but its e↵ect
on learning is di↵erent for these two components. The TD error (combined with eligibility
traces) tells the actor how to update action probabilities in order to reach higher-valued
states. Learning by the actor is like instrumental conditioning using a Law-of-E↵ect-type
learning rule (Section 1.7): the actor works to keep as positive as possible. On theother hand, the TD error (when combined with eligibility traces) tells the critic the
direction and magnitude in which to change the parameters of the value function in orderto improve its predictive accuracy. The critic works to reduce ’s magnitude to be as
close to zero as possible using a learning rule like the TD model of classical conditioning(Section 14.2). The di↵erence between the critic and actor learning rules is relatively
simple, but this di↵erence has a profound e↵ect on learning and is essential to how theactor–critic algorithm works. The di↵erence lies solely in the eligibility traces each typeof learning rule uses.
More than one set of learning rules can be used in actor–critic neural networks likethose in Figure 15.5b but, to be specific, here we focus on the actor–critic algorithm forcontinuing problems with eligibility traces presented in Section 13.6. On each transition
from state St to state St+1 , taking action At and receiving reward Rt+1 , that algorithm✓
computes the TD error ( ) and then updates the eligibility trace vectors (zwt and zt ) and15.8. Actor and Critic Learning Rules399

the parameters for the critic and actor (w and ✓), according tov̂(St ,w),
t = Rt+1 + v̂(St+1 ,w)
w w
w
zt =
zt 1 + rv̂(St ,w),
✓ ✓
✓
zt =
zt 1 + r ln ⇡(At |St , ✓),w
w + ↵ w t zw
t ,
✓
✓
✓
✓ + ↵ zt ,where 2 [0, 1) is a discount-rate parameter, w 2 [0, 1] and ✓ 2 [0, 1] are bootstrapping
parameters for the critic and the actor respectively, and ↵w > 0 and ↵✓ > 0 are analogous
step-size parameters.
Think of the approximate value function v̂ as the output of a single linear neuron-like
unit, called the critic unit and labeled V in Figure 15.5a. Then the value function is alinear function of the feature-vector representation of state s, x(s) = (x1 (s), . . . , xn (s))> ,parameterized by a weight vector w = (w1 , . . . , wn )> :v̂(s,w) = w> x(s).

(15.1)

Each xi (s) is like the presynaptic signal to a neuron’s synapse whose efficacy is wi . The
weights of the critic are incremented according to the rule above by ↵w t zw
t , where thereinforcement signal, t , corresponds to a dopamine signal being broadcast to all of thecritic unit’s synapses. The eligibility trace vector, zw
t , for the critic unit is a trace (averageof recent values) of rv̂(St ,w). Because v̂(s,w) is linear in the weights, rv̂(St ,w) = x(St ).
In neural terms, this means that each synapse has its own eligibility trace, which is
one component of the vector zw
t . A synapse’s eligibility trace accumulates according to
the level of activity arriving at that synapse, that is, the level of presynaptic activity,
represented here by the component of the feature vector x(St ) arriving at that synapse.
The trace otherwise decays toward zero at a rate governed by the fraction w . A synapse
is eligible for modification as long as its eligibility trace is non-zero. How the synapse’s
efficacy is actually modified depends on the reinforcement signals that arrive while thesynapse is eligible. We call eligibility traces like these of the critic unit’s synapses noncontingent eligibility traces because they only depend on presynaptic activity and are notcontingent in any way on postsynaptic activity.
The non-contingent eligibility traces of the critic unit’s synapses mean that the critic
unit’s learning rule is essentially the TD model of classical conditioning described in
Section 14.2. With the definition we have given above of the critic unit and its learning
rule, the critic in Figure 15.5a is the same as the critic in the ANN actor–critic of Bartoet al. (1983). Clearly, a critic like this consisting of just one linear neuron-like unit is thesimplest starting point; this critic unit is a proxy for a more complicated neural network
able to learn value functions of greater complexity.
The actor in Figure 15.5a is a one-layer network of k neuron-like actor units, each
receiving at time t the same feature vector, x(St ), that the critic unit receives. Eachactor unit j, j = 1, . . . , k, has its own weight vector, ✓j , but because the actor units are
all identical, we describe just one of the units and omit the subscript. One way for these

400

Chapter 15: Neuroscience

units to follow the actor–critic algorithm given in the equations above is for each to be a
Bernoulli-logistic unit. This means that the output of each actor unit at each time isa random variable, At , taking value 0 or 1. Think of value 1 as the neuron firing, thatis, emitting an action potential. The weighted sum, ✓ > x(St ), of a unit’s input vector
determines the unit’s action probabilities via the exponential soft-max distribution (13.2),
which for two actions is the logistic function:⇡(1|s, ✓) = 1⇡(0|s, ✓) =1
1 + exp( ✓ > x(s)).

(15.2)

The weights of each actor unit are incremented, as above, by: ✓✓ + ↵✓ t z✓t , whereagain corresponds to the dopamine signal: the same reinforcement signal that is sent to
all the critic unit’s synapses. Figure 15.5a shows t being broadcast to all the synapsesof all the actor units (which makes this actor network a team of reinforcement learning
agents, something we discuss in Section 15.10 below). The actor eligibility trace vectorz✓t is a trace (average of recent values) of r ln ⇡(At |St , ✓). To understand this eligibility
trace refer to Exercise 13.5, which defines this kind of unit and asks you to give a learning
rule for it. That exercise asked you to express r ln ⇡(a|s, ✓) in terms of a, x(s), and⇡(a|s, ✓) (for arbitrary state s and action a) by calculating the gradient. For the action
and state actually occurring at time t, the answer isr ln ⇡(At |St , ✓) = At⇡(1|St , ✓) x(St ).

(15.3)

Unlike the non-contingent eligibility trace of a critic synapse that only accumulates
the presynaptic activity x(St ), the eligibility trace of an actor unit’s synapse in addition
depends on the activity of the actor unit itself. We call this a contingent eligibility
trace because it is contingent on this postsynaptic activity. The eligibility trace at eachsynapse continually decays, but increments or decrements depending on the activity ofthe presynaptic neuron and whether or not the postsynaptic neuron fires. The factor
At ⇡(1|St , ✓) in (15.3) is positive when At = 1 and negative otherwise. The postsynaptic
contingency in the eligibility traces of actor units is the only di↵erence between the critic
and actor learning rules. By keeping information about what actions were taken inwhat states, contingent eligibility traces allow credit for reward (positive ), or blame forpunishment (negative ), to be apportioned among the policy parameters (the efficaciesof the actor units’ synapses) according to the contributions these parameters made to theunits’ outputs that could have influenced later values of . Contingent eligibility traces
mark the synapses as to how they should be modified to alter the units’ future responses
to favor positive values of .
What do the critic and actor learning rules suggest about how efficacies of corticostriatal
synapses change? Both learning rules are related to Donald Hebb’s classic proposal thatwhenever a presynaptic signal participates in activating the postsynaptic neuron, thesynapse’s efficacy increases (Hebb, 1949). The critic and actor learning rules share with
Hebb’s proposal the idea that changes in a synapse’s efficacy depend on the interactionof several factors. In the critic learning rule the interaction is between the reinforcement
signal and eligibility traces that depend only on presynaptic signals. Neuroscientists
call this a two-factor learning rule because the interaction is between two signals or15.8. Actor and Critic Learning Rules

401

quantities. The actor learning rule, on the other hand, is a three-factor learning rulebecause, in addition to depending on , its eligibility traces depend on both presynaptic
and postsynaptic activity. Unlike Hebb’s proposal, however, the relative timing of thefactors is critical to how synaptic efficacies change, with eligibility traces intervening toallow the reinforcement signal to a↵ect synapses that were active in the recent past.
Some subtleties about signal timing for the actor and critic learning rules deserve closer
attention. In defining the neuron-like actor and critic units, we ignored the small amountof time it takes synaptic input to e↵ect the firing of a real neuron. When an action
potential from the presynaptic neuron arrives at a synapse, neurotransmitter molecules
are released that di↵use across the synaptic cleft to the postsynaptic neuron, where
they bind to receptors on the postsynaptic neuron’s surface; this activates molecularmachinery that causes the postsynaptic neuron to fire (or to inhibit its firing in the case of
inhibitory synaptic input). This process can take several tens of milliseconds. According
to (15.1) and (15.2), though, the input to a critic and actor unit instantaneously produces
the unit’s output. Ignoring activation time like this is common in abstract models of
Hebbian-style plasticity in which synaptic efficacies change according to a simple productof simultaneous pre- and postsynaptic activity. More realistic models must take activation
time into account.
Activation time is especially important for a more realistic actor unit because it
influences how contingent eligibility traces have to work in order to properly apportion
credit for reinforcement to the appropriate synapses. The expression At ⇡(1|St , ✓) x(St )defining contingent eligibility traces for the actor unit’s learning rule given above includes
the postsynaptic factor At ⇡(1|St , ✓) and the presynaptic factor x(St ). This works
because by ignoring activation time, the presynaptic activity x(St ) participates in causing
the postsynaptic activity appearing in At ⇡(1|St , ✓) . To assign credit for reinforcement
correctly, the presynaptic factor defining the eligibility trace must be a cause of thepostsynaptic factor that also defines the trace. Contingent eligibility traces for a more
realistic actor unit would have to take activation time into account. (Activation timeshould not be confused with the time required for a neuron to receive a reinforcement
signal influenced by that neuron’s activity. The function of eligibility traces is to span
this time interval which is generally much longer than the activation time. We discuss
this further in the following section.)
There are hints from neuroscience for how this process might work in the brain.
Neuroscientists have discovered a form of Hebbian plasticity called spike-timing-dependent
plasticity (STDP) that lends plausibility to the existence of actor-like synaptic plasticity
in the brain. STDP is a Hebbian-style plasticity, but changes in a synapse’s efficacy
depend on the relative timing of presynaptic and postsynaptic action potentials. The
dependence can take di↵erent forms, but in the one most studied, a synapse increases instrength if spikes incoming via that synapse arrive shortly before the postsynaptic neuron
fires. If the timing relation is reversed, with a presynaptic spike arriving shortly after thepostsynaptic neuron fires, then the strength of the synapse decreases. STDP is a type of
Hebbian plasticity that takes the activation time of a neuron into account, which is oneof the ingredients needed for actor-like learning.

402

Chapter 15: Neuroscience

The discovery of STDP has led neuroscientists to investigate the possibility of a threefactor form of STDP in which neuromodulatory input must follow appropriately-timed
pre- and postsynaptic spikes. This form of synaptic plasticity, called reward-modulated
STDP, is much like the actor learning rule discussed here. Synaptic changes that would
be produced by regular STDP only occur if there is neuromodulatory input within a time
window after a presynaptic spike is closely followed by a postsynaptic spike. Evidence
is accumulating that reward-modulated STDP occurs at the spines of medium spiny
neurons of the dorsal striatum, with dopamine providing the neuromodulatory factor—
the sites where actor learning takes place in the hypothetical neural implementation ofan actor–critic algorithm illustrated in Figure 15.5b. Experiments have demonstrated
reward-modulated STDP in which lasting changes in the efficacies of corticostriatal
synapses occur only if a neuromodulatory pulse arrives within a time window that canlast up to 10 seconds after a presynaptic spike is closely followed by a postsynaptic
spike (Yagishita et al. 2014). Although the evidence is indirect, these experiments point
to the existence of contingent eligibility traces having prolonged time courses. The
molecular mechanisms producing these traces, as well as the much shorter traces thatlikely underly STDP, are not yet understood, but research focusing on time-dependentand neuromodulator-dependent synaptic plasticity is continuing.
The neuron-like actor unit that we have described here, with its Law-of-E↵ect-stylelearning rule, appeared in somewhat simpler form in the actor–critic network of Barto etal. (1983). That network was inspired by the “hedonistic neuron” hypothesis proposedby physiologist A. H. Klopf (1972, 1982). Not all the details of Klopf’s hypothesis are
consistent with what has been learned about synaptic plasticity, but the discovery of
STDP and the growing evidence for a reward-modulated form of STDP suggest that
Klopf’s ideas may not have been far o↵ the mark. We discuss Klopf’s hedonistic neuron
hypothesis next.

15.9

Hedonistic Neurons

In his hedonistic neuron hypothesis, Klopf (1972, 1982) conjectured that individual
neurons seek to maximize the di↵erence between synaptic input treated as rewarding
and synaptic input treated as punishing by adjusting the efficacies of their synapses
on the basis of rewarding or punishing consequences of their own action potentials. In
other words, individual neurons can be trained with response-contingent reinforcement
like an animal can be trained in an instrumental conditioning task. His hypothesis
included the idea that rewards and punishments are conveyed to a neuron via the same
synaptic input that excites or inhibits the neuron’s spike-generating activity. (Had Klopf
known what we know today about neuromodulatory systems, he might have assigned thereinforcing role to neuromodulatory input, but he wanted to avoid any centralized sourceof training information.) Synaptically-local traces of past pre- and postsynaptic activity
had the key function in Klopf’s hypothesis of making synapses eligible—the term heintroduced—for modification by later reward or punishment. He conjectured that thesetraces are implemented by molecular mechanisms local to each synapse and thereforedi↵erent from the electrical activity of both the pre- and the postsynaptic neurons. In

15.9. Hedonistic Neurons403

the Bibliographical and Historical Remarks section of this chapter we bring attention tosome similar proposals made by others.
Klopf specifically conjectured that synaptic efficacies change in the following way. Whena neuron fires an action potential, all of its synapses that were active in contributing tothat action potential become eligible to undergo changes in their efficacies. If the action
potential is followed within an appropriate time period by an increase of reward, theefficacies of all the eligible synapses increase. Symmetrically, if the action potential isfollowed within an appropriate time period by an increase of punishment, the efficacies
of eligible synapses decrease. This is implemented by triggering an eligibility trace at asynapse upon a coincidence of presynaptic and postsynaptic activity (or more exactly,
upon pairing of presynaptic activity with the postsynaptic activity that that presynaptic
activity participates in causing)—what we call a contingent eligibility trace. This isessentially the three-factor learning rule of an actor unit described in the previous section.
The shape and time course of an eligibility trace in Klopf’s theory reflects the durationsof the many feedback loops in which the neuron is embedded, some of which lie entirely
within the brain and body of the organism, while others extend out through the organism’s
external environment as mediated by its motor and sensory systems. His idea was thatthe shape of a synaptic eligibility trace is like a histogram of the durations of the feedback
loops in which the neuron is embedded. The peak of an eligibility trace would then occur
at the duration of the most prevalent feedback loops in which that neuron participates.
The eligibility traces used by algorithms described in this book are simplified versionsof Klopf’s original idea, being exponentially (or geometrically) decreasing functions
controlled by the parameters and . This simplifies simulations as well as theory, but
we regard these simple eligibility traces as a placeholders for traces closer to Klopf’s
original conception, which would have computational advantages in complex reinforcement
learning systems by refining the credit-assignment process.
Klopf’s hedonistic neuron hypothesis is not as implausible as it may at first appear.
A well-studied example of a single cell that seeks some stimuli and avoids others is thebacterium Escherichia coli. The movement of this single-cell organism is influenced bychemical stimuli in its environment, behavior known as chemotaxis. It swims in its liquid
environment by rotating hairlike structures called flagella attached to its surface. (Yes,it rotates them!) Molecules in the bacterium’s environment bind to receptors on its
surface. Binding events modulate the frequency with which the bacterium reverses flagellar
rotation. Each reversal causes the bacterium to tumble in place and then head o↵ in arandom new direction. A little chemical memory and computation causes the frequencyof flagellar reversal to decrease when the bacterium swims toward higher concentrationsof molecules it needs to survive (attractants) and increase when the bacterium swims
toward higher concentrations of molecules that are harmful (repellants). The result isthat the bacterium tends to persist in swimming up attractant gradients and tends to
avoid swimming up repellant gradients.
The chemotactic behavior just described is called klinokinesis. It is a kind of trialand-error behavior, although it is unlikely that learning is involved: the bacterium needsa modicum of short-term memory to detect molecular concentration gradients, but itprobably does not maintain long-term memories. Artificial intelligence pioneer Oliver404

Chapter 15: Neuroscience

Selfridge called this strategy “run and twiddle,” pointing out its utility as a basic adaptive
strategy: “keep going in the same way if things are getting better, and otherwise movearound” (Selfridge, 1978, 1984). Similarly, one might think of a neuron “swimming” (notliterally of course) in a medium composed of the complex collection of feedback loops
in which it is embedded, acting to obtain one type of input signal and to avoid others.
Unlike the bacterium, however, the neuron’s synaptic strengths retain information about
its past trial-and-error behavior. If this view of the behavior of a neuron (or just one typeof neuron) is plausible, then the closed-loop nature of how the neuron interacts with its
environment is important for understanding its behavior, where the neuron’s environment
consists of the rest of the animal together with the environment with which the animal
as a whole interacts.
Klopf’s hedonistic neuron hypothesis extended beyond the idea that individual neurons
are reinforcement learning agents. He argued that many aspects of intelligent behavior
can be understood as the result of the collective behavior of a population of self-interested
hedonistic neurons interacting with one another in an immense society or economic system
making up an animal’s nervous system. Whether or not this view of nervous systems
is useful, the collective behavior of reinforcement learning agents has implications forneuroscience. We take up this subject next.

15.10

Collective Reinforcement Learning

The behavior of populations of reinforcement learning agents is deeply relevant to thestudy of social and economic systems, and if anything like Klopf’s hedonistic neuron
hypothesis is correct, to neuroscience as well. The hypothesis described above about how
an actor–critic algorithm might be implemented in the brain only narrowly addresses
the implications of the fact that the dorsal and ventral subdivisions of the striatum, therespective locations of the actor and the critic according to the hypothesis, each containmillions of medium spiny neurons whose synapses undergo change modulated by phasic
bursts of dopamine neuron activity.
The actor in Figure 15.5a is a single-layer network of k actor units. The actions
produced by this network are vectors (A1 , A2 , · · · , Ak )> presumed to drive the animal’s
behavior. Changes in the efficacies of the synapses of all of these units depend on thereinforcement signal . Because actor units attempt to make as large as possible,e↵ectively acts as a reward signal for them (so in this case reinforcement is the sameas reward). Thus, each actor unit is itself a reinforcement learning agent—a hedonistic
neuron if you will. Now, to make the situation as simple as possible, assume that eachof these units receives the same reward signal at the same time (although, as indicatedabove, the assumption that dopamine is released at all the corticostriatal synapses under
the same conditions and at the same times is likely an oversimplification).
What can reinforcement learning theory tell us about what happens when all membersof a population of reinforcement learning agents learn according to a common reward
signal? The field of multi-agent reinforcement learning considers many aspects of learningby populations of reinforcement learning agents. Although this field is beyond the scopeof this book, we believe that some of its basic concepts and results are relevant to thinking

15.10.

Collective Reinforcement Learning405

about the brain’s di↵use neuromodulatory systems. In multi-agent reinforcement learning(and in game theory), the scenario in which all the agents try to maximize a common
reward signal that they simultaneously receive is known as a cooperative game or a teamproblem.
What makes a team problem interesting and challenging is that the common reward
signal sent to each agent evaluates the pattern of activity produced by the entire population,that is, it evaluates the collective action of the team members. This means that any
individual agent has only limited ability to a↵ect the reward signal because any single
agent contributes just one component of the collective action evaluated by the common
reward signal. E↵ective learning in this scenario requires addressing a structural credit
assignment problem: which team members, or groups of team members, deserve credit fora favorable reward signal, or blame for an unfavorable reward signal? It is a cooperative
game, or a team problem, because the agents are united in seeking to increase the samereward signal: there are no conflicts of interest among the agents. The scenario would be
a competitive game if di↵erent agents receive di↵erent reward signals, where each reward
signal again evaluates the collective action of the population, and the objective of eachagent is to increase its own reward signal. In this case there might be conflicts of interest
among the agents, meaning that actions that are good for some agents are bad for others.
Even deciding what the best collective action should be is a non-trivial aspect of gametheory. This competitive setting might be relevant to neuroscience too (for example, toaccount for heterogeneity of dopamine neuron activity), but here we focus only on thecooperative, or team, case.
How can each reinforcement learning agent in a team learn to “do the right thing” sothat the collective action of the team is highly rewarded? An interesting result is that
if each agent can learn e↵ectively despite its reward signal being corrupted by a large
amount of noise, and despite its lack of access to complete state information, then the
population as a whole will learn to produce collective actions that improve as evaluated by
the common reward signal, even when the agents cannot communicate with one another.
Each agent faces its own reinforcement learning task in which its influence on the reward
signal is deeply buried in the noise created by the influences of other agents. In fact, for
any agent, all the other agents are part of its environment because its input, both the partconveying state information and the reward part, depends on how all the other agentsare behaving. Furthermore, lacking access to the actions of the other agents, indeed
lacking access to the parameters determining their policies, each agent can only partiallyobserve the state of its environment. This makes each team member’s learning task very
difficult, but if each uses a reinforcement learning algorithm able to increase a reward
signal even under these difficult conditions, teams of reinforcement learning agents canlearn to produce collective actions that improve over time as evaluated by the team’scommon reward signal.
If the team members are neuron-like units, then each unit has to have the goal ofincreasing the amount of reward it receives over time, as the actor unit does that we
described in Section 15.8. Each unit’s learning algorithm has to have two essential features.
First, it has to use contingent eligibility traces. Recall that a contingent eligibility trace,in neural terms, is initiated (or increased) at a synapse when its presynaptic input

406

Chapter 15: Neuroscience

participates in causing the postsynaptic neuron to fire. A non-contingent eligibility trace,in contrast, is initiated or increased by presynaptic input independently of what thepostsynaptic neuron does. As explained in Section 15.8, by keeping information about
what actions were taken in what states, contingent eligibility traces allow credit for
reward, or blame for punishment, to be apportioned to an agent’s policy parameters
according to the contribution the values of these parameters made in determining theagent’s action. By similar reasoning, a team member must remember its recent action sothat it can either increase or decrease the likelihood of producing that action according
to the reward signal that is subsequently received. The action component of a contingent
eligibility trace implements this action memory. Because of the complexity of the learningtask, however, contingent eligibility is merely a preliminary step in the credit assignment
process: the relationship between a single team member’s action and changes in the
team’s reward signal is a statistical correlation that has to be estimated over many trials.
Contingent eligibility is an essential but preliminary step in this process.
Learning with non-contingent eligibility traces does not work at all in the team setting
because it does not provide a way to correlate actions with consequent changes in the
reward signal. Non-contingent eligibility traces are adequate for learning to predict, as
the critic component of the actor–critic algorithm does, but they do not support learning
to control, as the actor component must do. The members of a population of critic-likeagents may still receive a common reinforcement signal, but they would all learn topredict the same quantity (which in the case of an actor–critic method, would be theexpected return for the current policy). How successful each member of the population
would be in learning to predict the expected return would depend on the information it
receives, which could be very di↵erent for di↵erent members of the population. There
would be no need for the population to produce di↵erentiated patterns of activity. This
is not a team problem as defined here.
A second requirement for collective learning in a team problem is that there has to bevariability in the actions of the team members in order for the team to explore the spaceof collective actions. The simplest way for a team of reinforcement learning agents to do
this is for each member to independently explore its own action space through persistentvariability in its output. This will cause the team as a whole to vary its collective actions.
For example, a team of the actor units described in Section 15.8 explores the spaceof collective actions because the output of each unit, being a Bernoulli-logistic unit,
probabilistically depends on the weighted sum of its input vector’s components. The
weighted sum biases firing probability up or down, but there is always variability. Because
each unit uses a REINFORCE policy gradient algorithm (Chapter 13), each unit adjusts
its weights with the goal of maximizing the average reward rate it experiences while
stochastically exploring its own action space. One can show, as Williams (1992) did, thata team of Bernoulli-logistic REINFORCE units implements a policy gradient algorithm
as a whole with respect to average rate of the team’s common reward signal, where theactions are the collective actions of the team.
Further, Williams (1992) showed that a team of Bernoulli-logistic units using REINFORCE ascends the average reward gradient when the units in the team are interconnected
to form a multilayer ANN. In this case, the reward signal is broadcast to all the units in15.11.

Model-based Methods in the Brain407

the network, though reward may depend only on the collective actions of the network’s
output units. This means that a multilayer team of Bernoulli-logistic REINFORCE
units learns like a multilayer network trained by the widely-used error backpropagation
method, but in this case the backpropagation process is replaced by the broadcasted
reward signal. In practice, the error backpropagation method is considerably faster,
but the reinforcement learning team method is more plausible as a neural mechanism,
especially in light of what is being learned about reward-modulated STDP as discussed
in Section 15.8.
Exploration through independent exploration by team members is only the simplest
way for a team to explore; more sophisticated methods are possible if the team members
coordinate their actions to focus on particular parts of the collective action space, either
by communicating with one another or by responding to common inputs. There are alsomechanisms more sophisticated than contingent eligibility traces for addressing structural
credit assignment, which is easier in a team problem when the set of possible collective
actions is restricted in some way. An extreme case is a winner-take-all arrangement (forexample, the result of lateral inhibition in the brain) that restricts collective actions tothose to which only one, or a few, team members contribute. In this case the winners get
the credit or blame for resulting reward or punishment.
Details of learning in cooperative games (or team problems) and non-cooperative
game problems are beyond the scope of this book. The Bibliographical and Historical
Remarks section at the end of this chapter cites a selection of the relevant publications,
including extensive references to research on implications for neuroscience of collective
reinforcement learning.

15.11

Model-based Methods in the Brain

Reinforcement learning’s distinction between model-free and model-based algorithms isproving to be useful for thinking about animal learning and decision processes. Section 14.6
discusses how this distinction aligns with that between habitual and goal-directed animal
behavior. The hypothesis discussed above about how the brain might implement anactor–critic algorithm is relevant only to an animal’s habitual mode of behavior because
the basic actor–critic method is model-free. What neural mechanisms are responsible
for producing goal-directed behavior, and how do they interact with those underlying
habitual behavior?
One way to investigate questions about the brain structures involved in these modesof behavior is to inactivate an area of a rat’s brain and then observe what the rat does inan outcome-devaluation experiment (Section 14.6). Results from experiments like these
indicate that the actor–critic hypothesis described above is too simple in placing theactor in the dorsal striatum. Inactivating one part of the dorsal striatum, the dorsolateral
striatum (DLS), impairs habit learning, causing the animal to rely more on goal-directed
processes. On the other hand, inactivating the dorsomedial striatum (DMS) impairs
goal-directed processes, requiring the animal to rely more on habit learning. Results
like these support the view that the DLS in rodents is more involved in model-freeprocesses, whereas their DMS is more involved in model-based processes. Results of408

Chapter 15: Neurosciencestudies with human subjects in similar experiments using functional neuroimaging, andwith non-human primates, support the view that the analogous structures in the primate
brain are di↵erentially involved in habitual and goal-directed modes of behavior.
Other studies identify activity associated with model-based processes in the prefrontal
cortex of the human brain, the front-most part of the frontal cortex implicated in
executive function, including planning and decision making. Specifically implicated is theorbitofrontal cortex (OFC), the part of the prefrontal cortex immediately above the eyes.
Functional neuroimaging in humans, and also recordings of the activities of single neurons
in monkeys, reveals strong activity in the OFC related to the subjective reward valueof biologically significant stimuli, as well as activity related to the reward expected as aconsequence of actions. Although not free of controversy, these results suggest significant
involvement of the OFC in goal-directed choice. It may be critical for the reward part ofan animal’s environment model.
Another structure involved in model-based behavior is the hippocampus, a structure
critical for memory and spatial navigation. A rat’s hippocampus plays a critical rolein the rat’s ability to navigate a maze in the goal-directed manner that led Tolman to
the idea that animals use models, or cognitive maps, in selecting actions (Section 14.5).
The hippocampus may also be a critical component of our human ability to imagine
new experiences (Hassabis and Maguire, 2007; Ólafsdóttir, Barry, Saleem, Hassabis, andSpiers, 2015).
The findings that most directly implicate the hippocampus in planning—the processneeded to enlist an environment model in making decisions—come from experimentsthat decode the activity of neurons in the hippocampus to determine what part of spacehippocampal activity is representing on a moment-to-moment basis. When a rat pauses
at a choice point in a maze, the representation of space in the hippocampus sweeps
forward (and not backwards) along the possible paths the animal can take from thatpoint (Johnson and Redish, 2007). Furthermore, the spatial trajectories represented bythese sweeps closely correspond to the rat’s subsequent navigational behavior (Pfei↵erand Foster, 2013). These results suggest that the hippocampus is critical for the statetransition part of an animal’s environment model, and that it is part of a system thatuses the model to simulate possible future state sequences to assess the consequences ofpossible courses of action: a form of planning.
The results described above add to a voluminous literature on neural mechanisms
underlying goal-directed, or model-based, learning and decision making, but manyquestions remain unanswered. For example, how can areas as structurally similar as the
DLS and DMS be essential components of modes of learning and behavior that are asdi↵erent as model-free and model-based algorithms? Are separate structures responsiblefor (what we call) the transition and reward components of an environment model? Is allplanning conducted at decision time via simulations of possible future courses of action
as the forward sweeping activity in the hippocampus suggests? In other words, is allplanning something like a rollout algorithm (Section 8.10)? Or are models sometimes
engaged in the background to refine or recompute value information as illustrated by the
Dyna architecture (Section 8.2)? How does the brain arbitrate between the use of thehabit and goal-directed systems? Is there, in fact, a clear separation between the neuralsubstrates of these systems?

15.12.

Addiction

409

The evidence is not pointing to a positive answer to this last question. Summarizing
the situation, Doll, Simon, and Daw (2012) wrote that “model-based influences appear
ubiquitous more or less wherever the brain processes reward information,” and this istrue even in the regions thought to be critical for model-free learning. This includes the
dopamine signals themselves, which can exhibit the influence of model-based informationin addition to the reward prediction errors thought to be the basis of model-free processes.
Continuing neuroscience research informed by reinforcement learning’s model-free andmodel-based distinction has the potential to sharpen our understanding of habitual andgoal-directed processes in the brain. A better grasp of these neural mechanisms may leadto algorithms combining model-free and model-based methods in ways that have not yet
been explored in computational reinforcement learning.

15.12

Addiction

Understanding the neural basis of drug abuse is a high-priority goal of neuroscience withthe potential to produce new treatments for this serious public health problem. One
view is that drug craving is the result of the same motivation and learning processes thatlead us to seek natural rewarding experiences that serve our biological needs. Addictive
substances, by being intensely reinforcing, e↵ectively co-opt our natural mechanismsof learning and decision making. This is plausible given that many—though not all—drugs of abuse increase levels of dopamine either directly or indirectly in regions aroundterminals of dopamine neuron axons in the striatum, a brain structure firmly implicated innormal reward-based learning (Section 15.7). But the self-destructive behavior associatedwith drug addiction is not characteristic of normal learning. What is di↵erent about
dopamine-mediated learning when the reward is the result of an addictive drug? Is
addiction the result of normal learning in response to substances that were largely
unavailable throughout our evolutionary history, so that evolution could not select against
their damaging e↵ects? Or do addictive substances somehow interfere with normal
dopamine-mediated learning?
The reward prediction error hypothesis of dopamine neuron activity and its connection
to TD learning are the basis of a model due to Redish (2004) of some—but certainly not
all—features of addiction. The model is based on the observation that administration of
cocaine and some other addictive drugs produces a transient increase in dopamine. In themodel, this dopamine surge is assumed to increase the TD error, , in a way that cannot
be cancelled out by changes in the value function. In other words, whereas is reduced
to the degree that a normal reward is predicted by antecedent events (Section 15.6), thecontribution to due to an addictive stimulus does not decrease as the reward signal
becomes predicted: drug rewards cannot be “predicted away.” The model does this bypreventing from ever becoming negative when the reward signal is due to an addictive
drug, thus eliminating the error-correcting feature of TD learning for states associatedwith administration of the drug. The result is that the values of these states increase
without bound, making actions leading to these states preferred above all others.

410

Chapter 15: Neuroscience

Addictive behavior is much more complicated than this result from Redish’s model, but
the model’s main idea may be a piece of the puzzle. Or the model might be misleading.
Dopamine appears not to play a critical role in all forms of addiction, and not everyone
is equally susceptible to developing addictive behavior. Moreover, the model does notinclude the changes in many circuits and brain regions that accompany chronic drug
taking, for example, changes that lead to a drug’s diminishing e↵ect with repeated use.
It is also likely that addiction involves model-based processes. Still, Redish’s model
illustrates how reinforcement learning theory can be enlisted in the e↵ort to understanda major health problem. In a similar manner, reinforcement learning theory has been
influential in the development of the new field of computational psychiatry, which aims
to improve understanding of mental disorders through mathematical and computational
methods.

15.13

Summary

The neural pathways involved in the brain’s reward system are complex and incompletely
understood, but neuroscience research directed toward understanding these pathways
and their roles in behavior is progressing rapidly. This research is revealing striking
correspondences between the brain’s reward system and the theory of reinforcement
learning as presented in this book.
The reward prediction error hypothesis of dopamine neuron activity was proposed byscientists who recognized striking parallels between the behavior of TD errors and theactivity of neurons that produce dopamine, a neurotransmitter essential in mammals
for reward-related learning and behavior. Experiments conducted in the late 1980s and1990s in the laboratory of neuroscientist Wolfram Schultz showed that dopamine neurons
respond to rewarding events with substantial bursts of activity, called phasic responses,
only if the animal does not expect those events, suggesting that dopamine neurons are
signaling reward prediction errors instead of reward itself. Further, these experiments
showed that as an animal learns to predict a rewarding event on the basis of preceding
sensory cues, the phasic activity of dopamine neurons shifts to earlier predictive cues
while decreasing to later predictive cues. This parallels the backing-up e↵ect of the TD
error as a reinforcement learning agent learns to predict reward.
Other experimental results firmly establish that the phasic activity of dopamine neurons
is a reinforcement signal for learning that reaches multiple areas of the brain by means ofprofusely branching axons of dopamine producing neurons. These results are consistent
with the distinction we make between a reward signal, Rt , and a reinforcement signal,
which is the TD error t in most of the algorithms we present. Phasic responses of
dopamine neurons are reinforcement signals, not reward signals.
A prominent hypothesis is that the brain implements something like an actor–critic
algorithm. Two structures in the brain (the dorsal and ventral subdivisions of thestriatum), both of which play critical roles in reward-based learning, may function
respectively like an actor and a critic. That the TD error is the reinforcement signal forboth the actor and the critic fits well with the facts that dopamine neuron axons target
both the dorsal and ventral subdivisions of the striatum; that dopamine appears to be15.13.

Summary

411

critical for modulating synaptic plasticity in both structures; and that the e↵ect on atarget structure of a neuromodulator such as dopamine depends on properties of thetarget structure and not just on properties of the neuromodulator.
The actor and the critic can be implemented by ANNs consisting of neuron-like units
having learning rules based on the policy-gradient actor–critic method described in
Section 13.5. Each connection in these networks is like a synapse between neurons in
the brain, and the learning rules correspond to rules governing how synaptic efficacies
change as functions of the activities of the presynaptic and the postsynaptic neurons,
together with neuromodulatory input corresponding to input from dopamine neurons. In
this setting, each synapse has its own eligibility trace that records past activity involving
that synapse. The only di↵erence between the actor and critic learning rules is that theyuse di↵erent kinds of eligibility traces: the critic unit’s traces are non-contingent because
they do not involve the critic unit’s output, whereas the actor unit’s traces are contingent
because in addition to the actor unit’s input, they depend on the actor unit’s output. In
the hypothetical implementation of an actor–critic system in the brain, these learning
rules respectively correspond to rules governing plasticity of corticostriatal synapses thatconvey signals from the cortex to the principal neurons in the dorsal and ventral striatal
subdivisions, synapses that also receive inputs from dopamine neurons.
The learning rule of an actor unit in the actor–critic network closely corresponds toreward-modulated spike-timing-dependent plasticity. In spike-timing-dependent plasticity(STDP), the relative timing of pre- and postsynaptic activity determines the direction ofsynaptic change. In reward-modulated STDP, changes in synapses in addition depend
on a neuromodulator, such as dopamine, arriving within a time window that can last
up to 10 seconds after the conditions for STDP are met. Evidence is accumulating thatreward-modulated STDP occurs at corticostriatal synapses, where the actor’s learning
takes place in the hypothetical neural implementation of an actor–critic system, adds tothe plausibility of the hypothesis that something like an actor–critic system exists in thebrains of some animals.
The idea of synaptic eligibility and basic features of the actor learning rule derivefrom Klopf’s hypothesis of the “hedonistic neuron” (Klopf, 1972, 1981). He conjectured
that individual neurons seek to obtain reward and to avoid punishment by adjusting theefficacies of their synapses on the basis of rewarding or punishing consequences of theiraction potentials. A neuron’s activity can a↵ect its later input because the neuron isembedded in many feedback loops, some within the animal’s nervous system and bodyand others passing through the animal’s external environment. Klopf’s idea of eligibility
is that synapses are temporarily marked as eligible for modification if they participatedin the neuron’s firing (making this the contingent form of eligibility trace). A synapse’s
efficacy is modified if a reinforcing signal arrives while the synapse is eligible. We alluded
to the chemotactic behavior of a bacterium as an example of a single cell that directs its
movements in order to seek some molecules and to avoid others.
A conspicuous feature of the dopamine system is that fibers releasing dopamine project
widely to multiple parts of the brain. Although it is likely that only some populationsof dopamine neurons broadcast the same reinforcement signal, if this signal reaches
the synapses of many neurons involved in actor-type learning, then the situation can

412

Chapter 15: Neurosciencebe modeled as a team problem. In this type of problem, each agent in a collection ofreinforcement learning agents receives the same reinforcement signal, where that signal
depends on the activities of all members of the collection, or team. If each team member
uses a sufficiently capable learning algorithm, the team can learn collectively to improve
performance of the entire team as evaluated by the globally-broadcast reinforcement
signal, even if the team members do not directly communicate with one another. Thisis consistent with the wide dispersion of dopamine signals in the brain and provides
a neurally plausible alternative to the widely-used error-backpropagation method fortraining multilayer networks.
The distinction between model-free and model-based reinforcement learning is helping
neuroscientists investigate the neural bases of habitual and goal-directed learning anddecision making. Research so far points to their being some brain regions more involvedin one type of process than the other, but the picture remains unclear because model-freeand model-based processes do not appear to be neatly separated in the brain. Many
questions remain unanswered. Perhaps most intriguing is evidence that the hippocampus,
a structure traditionally associated with spatial navigation and memory, appears to be
involved in simulating possible future courses of action as part of an animal’s decisionmaking process. This suggests that it is part of a system that uses an environment model
for planning.
Reinforcement learning theory is also influencing thinking about neural processes
underlying drug abuse. A model of some features of drug addiction is based on the reward
prediction error hypothesis. It proposes that an addicting stimulant, such as cocaine,destabilizes TD learning to produce unbounded growth in the values of actions associatedwith drug intake. This is far from a complete model of addiction, but it illustrates how
a computational perspective suggests theories that can be tested with further research.
The new field of computational psychiatry similarly focuses on the use of computationalmodels, some derived from reinforcement learning, to better understand mental disorders.
This chapter only touched the surface of how the neuroscience of reinforcement learningand the development of reinforcement learning in computer science and engineering
have influenced one another. Most features of reinforcement learning algorithms owe
their design to purely computational considerations, but some have been influenced byhypotheses about neural learning mechanisms. Remarkably, as experimental data has
accumulated about the brain’s reward processes, many of the purely computationallymotivated features of reinforcement learning algorithms are turning out to be consistent
with neuroscience data. Other features of computational reinforcement learning, such as
eligibility traces and the ability of teams of reinforcement learning agents to learn to act
collectively under the influence of a globally-broadcast reinforcement signal, may also
turn out to parallel experimental data as neuroscientists continue to unravel the neuralbasis of reward-based animal learning and behavior.

15.13.

Summary

413

Bibliographical and Historical Remarks
The number of publications treating parallels between the neuroscience of learning anddecision making and the approach to reinforcement learning presented in this book isenormous. We can cite only a small selection. Niv (2009), Dayan and Niv (2008), Gimcher(2011), Ludvig, Bellemare, and Pearson (2011), and Shah (2012) are good places to start.
Together with economics, evolutionary biology, and mathematical psychology, reinforcement learning theory is helping to formulate quantitative models of the neural mechanismsof choice in humans and non-human primates. With its focus on learning, this chapter
only lightly touches upon the neuroscience of decision making. Glimcher (2003) introduced
the field of “neuroeconomics,” in which reinforcement learning contributes to the studyof the neural basis of decision making from an economics perspective. See also Glimcher
and Fehr (2013). The text on computational and mathematical modeling in neuroscienceby Dayan and Abbott (2001) includes reinforcement learning’s role in these approaches.
Sterling and Laughlin (2015) examined the neural basis of learning in terms of generaldesign principles that enable efficient adaptive behavior.
15.1

There are many good expositions of basic neuroscience. Kandel, Schwartz, Jessell,
Siegelbaum, and Hudspeth (2013) is an authoritative and very comprehensive
source.

15.2

Berridge and Kringelbach (2008) reviewed the neural basis of reward and pleasure,pointing out that reward processing has many dimensions and involves many
neural systems. Space prevents discussion of the influential research of Berridge
and Robinson (1998), who distinguish between the hedonic impact of a stimulus,
which they call “liking,” and the motivational e↵ect, which they call “wanting.”
Hare, O’Doherty, Camerer, Schultz, and Rangel (2008) examined the neural basisof value-related signals from an economic perspective, distinguishing between
goal values, decision values, and prediction errors. Decision value is goal value
minus action cost. See also Rangel, Camerer, and Montague (2008), Rangel andHare (2010), and Peters and Büchel (2010).

15.3

The reward prediction error hypothesis of dopamine neuron activity is mostprominently discussed by Schultz, Dayan, and Montague (1997). The hypothesis
was first explicitly put forward by Montague, Dayan, and Sejnowski (1996). As
they stated the hypothesis, it referred to reward prediction errors (RPEs) but
not specifically to TD errors; however, their development of the hypothesis madeit clear that they were referring to TD errors. The earliest recognition of the
TD-error/dopamine connection of which we are aware is that of Montague, Dayan,
Nowlan, Pouget, and Sejnowski (1993), who proposed a TD-error-modulated
Hebbian learning rule motivated by results on dopamine signaling from Schultz’sgroup. The connection was also pointed out in an abstract by Quartz, Dayan,
Montague, and Sejnowski (1992). Montague and Sejnowski (1994) emphasized
the importance of prediction in the brain and outlined how predictive Hebbianlearning modulated by TD errors could be implemented via a di↵use neuromodulatory system, such as the dopamine system. Friston, Tononi, Reeke, Sporns,

414

Chapter 15: Neuroscienceand Edelman (1994) presented a model of value-dependent learning in the brain
in which synaptic changes are mediated by a TD-like error provided by a global
neuromodulatory signal (although they did not single out dopamine). Montague, Dayan, Person, and Sejnowski (1995) presented a model of honeybee
foraging using the TD error. The model is based on research by Hammer, Menzel, and colleagues (Hammer and Menzel, 1995; Hammer, 1997) showing thatthe neuromodulator octopamine acts as a reinforcement signal in the honeybee.
Montague et al. (1995) pointed out that dopamine likely plays a similar rolein the vertebrate brain. Barto (1995a) related the actor–critic architecture tobasal-ganglionic circuits and discussed the relationship between TD learningand the main results from Schultz’s group. Houk, Adams, and Barto (1995)
suggested how TD learning and the actor–critic architecture might map onto theanatomy, physiology, and molecular mechanism of the basal ganglia. Doya andSejnowski (1998) extended their earlier paper on a model of birdsong learning(Doya and Sejnowski, 1995) by including a TD-like error identified with dopamine
to reinforce the selection of auditory input to be memorized. O’Reilly and Frank(2006) and O’Reilly, Frank, Hazy, and Watz (2007) argued that phasic dopamine
signals are RPEs but not TD errors. In support of their theory they cited resultswith variable interstimulus intervals that do not match predictions of a simple
TD model, as well as the observation that higher-order conditioning beyond
second-order conditioning is rarely observed, while TD learning is not so limited.
Dayan and Niv (2008) discussed “the good, the bad, and the ugly” of how
reinforcement learning theory and the reward prediction error hypothesis alignwith experimental data. Glimcher (2011) reviewed the empirical findings that
support the reward prediction error hypothesis and emphasized the significanceof the hypothesis for contemporary neuroscience.
15.4

Graybiel (2000) is a brief primer on the basal ganglia. The experiments mentionedthat involve optogenetic activation of dopamine neurons were conducted by Tsai,
Zhang, Adamantidis, Stuber, Bonci, de Lecea, and Deisseroth (2009), Steinberg,
Keiflin, Boivin, Witten, Deisseroth, and Janak (2013), and Claridge-Chang,Roorda, Vrontou, Sjulson, Li, Hirsh, and Miesenböck (2009). Fiorillo, Yun, andSong (2013), Lammel, Lim, and Malenka (2014), and Saddoris, Cacciapaglia,
Wightmman, and Carelli (2015) are among studies showing that the signaling
properties of dopamine neurons are specialized for di↵erent target regions. RPEsignaling neurons may belong to one among multiple populations of dopamine
neurons having di↵erent targets and subserving di↵erent functions. Eshel, Tian,
Bukwich, and Uchida (2016) found homogeneity of reward prediction error
responses of dopamine neurons in the lateral VTA during classical conditioningin mice, though their results do not rule out response diversity across wider areas.
Gershman, Pesaran, and Daw (2009) studied reinforcement learning tasks thatcan be decomposed into independent components with separate reward signals,
finding evidence in human neuroimaging data suggesting that the brain exploits
this kind of structure.

15.13.

Summary415

15.5

Schultz’s 1998 survey article is a good entrée into the very extensive literatureon reward predicting signaling of dopamine neurons. Berns, McClure, Pagnoni,
and Montague (2001), Breiter, Aharon, Kahneman, Dale, and Shizgal (2001),
Pagnoni, Zink, Montague, and Berns (2002), and O’Doherty, Dayan, Friston,
Critchley, and Dolan (2003) described functional brain imaging studies supporting
the existence of signals like TD errors in the human brain.

15.6

This section roughly follows Barto (1995a) in explaining how TD errors mimic themain results from Schultz’s group on the phasic responses of dopamine neurons.

15.7

This section is largely based on Takahashi, Schoenbaum, and Niv (2008) and Niv(2009). To the best of our knowledge, Barto (1995a) and Houk, Adams, and Barto(1995) first speculated about possible implementations of actor–critic algorithms
in the basal ganglia. On the basis of functional magnetic resonance imaging of
human subjects while engaged in instrumental conditioning, O’Doherty, Dayan,
Schultz, Deichmann, Friston, and Dolan (2004) suggested that the actor andthe critic are most likely located respectively in the dorsal and ventral striatum.
Gershman, Moustafa, and Ludvig (2014) focused on how time is represented inreinforcement learning models of the basal ganglia, discussing evidence for, andimplications of, various computational approaches to time representation.
The hypothetical neural implementation of the actor–critic architecture describedin this section includes very little detail about known basal ganglia anatomy andphysiology. In addition to the more detailed hypothesis of Houk, Adams, andBarto (1995), a number of other hypotheses include more specific connections toanatomy and physiology and are claimed to explain additional data. These include
hypotheses proposed by Suri and Schultz (1998, 1999), Brown, Bullock, andGrossberg (1999), Contreras-Vidal and Schultz (1999), Suri, Bargas, and Arbib(2001), O’Reilly and Frank (2006), and O’Reilly, Frank, Hazy, and Watz (2007).
Joel, Niv, and Ruppin (2002) critically evaluated the anatomical plausibility ofseveral of these models and present an alternative intended to accommodate
some neglected features of basal ganglionic circuitry.

15.8

The actor learning rule discussed here is more complicated than the one in theearly actor–critic network of Barto et al. (1983). Actor-unit eligibility traces inthat network were traces of just At ⇥x(St ) instead of the full (At ⇡(1|St , ✓))x(St ).
That work did not benefit from the policy-gradient theory presented in Chapter 13
or the contributions of Williams (1986, 1992), who showed how an ANN ofBernoulli-logistic units could implement a policy-gradient method.
Reynolds and Wickens (2002) proposed a three-factor rule for synaptic plasticity
in the corticostriatal pathway in which dopamine modulates changes in corticostriatal synaptic efficacy. They discussed the experimental support for thiskind of learning rule and its possible molecular basis. The definitive demonstration of spike-timing-dependent plasticity (STDP) is attributed to Markram,
Lübke, Frotscher, and Sakmann (1997), with evidence from earlier experiments

416

Chapter 15: Neuroscienceby Levy and Steward (1983) and others that the relative timing of pre- andpostsynaptic spikes is critical for inducing changes in synaptic efficacy. Rao andSejnowski (2001) suggested how STDP could be the result of a TD-like mechanism
at synapses with non-contingent eligibility traces lasting about 10 milliseconds.
Dayan (2002) commented that this would require an error as in Sutton andBarto’s (1981a) early model of classical conditioning and not a true TD error.
Representative publications from the extensive literature on reward-modulated
STDP are Wickens (1990), Reynolds and Wickens (2002), and Calabresi, Picconi,
Tozzi and Di Filippo (2007). Pawlak and Kerr (2008) showed that dopamine isnecessary to induce STDP at the corticostriatal synapses of medium spiny neurons.
See also Pawlak, Wickens, Kirkwood, and Kerr (2010). Yagishita, Hayashi-Takagi,
Ellis-Davies, Urakubo, Ishii, and Kasai (2014) found that dopamine promotes
spine enlargement of the medium spiny neurons of mice only during a time windowof from 0.3 to 2 seconds after STDP stimulation. Izhikevich (2007) proposedand explored the idea of using STDP timing conditions to trigger contingent
eligibility traces. Frémaux, Sprekeler, and Gerstner (2010) proposed theoreticalconditions for successful learning by rules based on reward-modulated STDP.
15.9

Klopf’s hedonistic neuron hypothesis (Klopf 1972, 1982) inspired our actor–critic
algorithm implemented as an ANN with a single neuron-like unit, called theactor unit, implementing a Law-of-E↵ect-like learning rule (Barto, Sutton, andAnderson, 1983). Ideas related to Klopf’s synaptically-local eligibility have been
proposed by others. Crow (1968) proposed that changes in the synapses ofcortical neurons are sensitive to the consequences of neural activity. Emphasizing
the need to address the time delay between neural activity and its consequences
in a reward-modulated form of synaptic plasticity, he proposed a contingent formof eligibility, but associated with entire neurons instead of individual synapses.
According to his hypothesis, a wave of neuronal activity
leads to a short-term change in the cells involved in the wave such that they
are picked out from a background of cells not so activated. ... such cells are
rendered sensitive by the short-term change to a reward signal ... in sucha way that if such a signal occurs before the end of the decay time of thechange the synaptic connexions between the cells are made more e↵ective.
(Crow, 1968)

Crow argued against previous proposals that reverberating neural circuits playthis role by pointing out that the e↵ect of a reward signal on such a circuit would“...establish the synaptic connexions leading to the reverberation (that is to say,those involved in activity at the time of the reward signal) and not those on thepath which led to the adaptive motor output.” Crow further postulated thatreward signals are delivered via a “distinct neural fiber system,” presumably the
one into which Olds and Milner (1954) tapped, that would transform synaptic
connections “from a short into a long-term form.”

15.13.

Summary417

In another farsighted hypothesis, Miller proposed a Law-of-E↵ect-like learning
rule that includes synaptically-local contingent eligibility traces:... it is envisaged that in a particular sensory situation neurone B, by chance,
fires a ‘meaningful burst’ of activity, which is then translated into motor acts,
which then change the situation. It must be supposed that the meaningful
burst has an influence, at the neuronal level, on all of its own synapses which
are active at the time ... thereby making a preliminary selection of thesynapses to be strengthened, though not yet actually strengthening them.
...The strengthening signal ... makes the final selection ... and accomplishes
the definitive change in the appropriate synapses. (Miller, 1981, p. 81)

Miller’s hypothesis also included a critic-like mechanism, which he called a “sensory analyzer unit,” that worked according to classical conditioning principles toprovide reinforcement signals to neurons so that they would learn to move fromlower- to higher-valued states, thus anticipating the use of the TD error as a reinforcement signal in the actor–critic architecture. Miller’s idea not only parallels
Klopf’s (with the exception of its explicit invocation of a distinct “strengthening
signal”), it also anticipated the general features of reward-modulated STDP.
A related though di↵erent idea, which Seung (2003) called the “hedonistic
synapse,” is that synapses individually adjust the probability that they releaseneurotransmitter in the manner of the Law of E↵ect: if reward follows release,
the release probability increases, and decreases if reward follows failure to release.
This is essentially the same as the learning scheme Minsky used in his 1954
Princeton PhD dissertation, where he called the synapse-like learning elementa SNARC (Stochastic Neural-Analog Reinforcement Calculator). Contingent
eligibility is involved in these ideas too, although it is contingent on the activityof an individual synapse instead of the postsynaptic neuron. Also related is the
proposal of Unnikrishnan and Venugopal (1994) that uses the correlation-based
method of Harth and Tzanakou (1974) to adjust ANN weights.
Frey and Morris (1997) proposed the idea of a “synaptic tag” for the inductionof long-lasting strengthening of synaptic efficacy. Though not unlike Klopf’s
eligibility, their tag was hypothesized to consist of a temporary strengthening of asynapse that could be transformed into a long-lasting strengthening by subsequentneuron activation. The model of O’Reilly and Frank (2006) and O’Reilly, Frank,
Hazy, and Watz (2007) uses working memory to bridge temporal intervals insteadof eligibility traces. Wickens and Kotter (1995) discuss possible mechanisms
for synaptic eligibility. He, Huertas, Hong, Tie, Hell, Shouval, Kirkwood (2015)
provide evidence supporting the existence of contingent eligibility traces insynapses of cortical neurons with time courses like those of the eligibility traces
Klopf postulated.
The metaphor of a neuron using a learning rule related to bacterial chemotaxis was
discussed by Barto (1989). Koshland’s extensive study of bacterial chemotaxis
was in part motivated by similarities between features of bacteria and features ofneurons (Koshland, 1980). See also Berg (1975). Shimansky (2009) proposed a418

Chapter 15: Neuroscience

synaptic learning rule somewhat similar to Seung’s mentioned above in which
each synapse individually acts like a chemotactic bacterium. In this case acollection of synapses “swims” toward attractants in the high-dimensional spaceof synaptic weight values. Montague, Dayan, Person, and Sejnowski (1995)
proposed a chemotactic-like model of the bee’s foraging behavior involving theneuromodulator octopamine.
15.10 Research on the behavior of reinforcement learning agents in team and gameproblems has a long history roughly occurring in three phases. To the bestof our knowledge, the first phase began with investigations by the Russianmathematician and physicist M. L. Tsetlin. A collection of his work was publishedas Tsetlin (1973) after his death in 1966. Our Sections 1.7 and 4.8 refer to his
study of learning automata in connection to bandit problems. The Tsetlincollection also includes studies of learning automata in team and game problems,
which led to later work in this area using stochastic learning automata as
described by Narendra and Thathachar (1974, 1989), Viswanathan and Narendra(1974), Lakshmivarahan and Narendra (1982), Narendra and Wheeler (1983), andThathachar and Sastry (2002). Thathachar and Sastry (2011) is a more recent
comprehensive account. These studies were mostly restricted to non-associative
learning automata, meaning that they did not address associative, or contextual,
bandit problems (Section 2.9).
The second phase began with the extension of learning automata to the associative,
or contextual, case. Barto, Sutton, and Brouwer (1981) and Barto and Sutton(1981b) experimented with associative stochastic learning automata in singlelayer ANNs to which a global reinforcement signal was broadcast. The learningalgorithm was an associative extension of the Alopex algorithm of Harth andTzanakou (1974). Barto et al. called neuron-like elements implementing thiskind of learning associative search elements (ASEs). Barto and Anandan (1985)
introduced an associative reinforcement learning algorithm called the associative
reward-penalty (AR P ) algorithm. They proved a convergence result by combining
theory of stochastic learning automata with theory of pattern classification. Barto(1985, 1986) and Barto and Jordan (1987) described results with teams of AR P
units connected into multi-layer ANNs, showing that they could learn nonlinear
functions, such as XOR and others, with a globally-broadcast reinforcement signal.
Barto (1985) extensively discussed this approach to ANNs and how this type oflearning rule is related to others in the literature at that time. Williams (1992)
mathematically analyzed and broadened this class of learning rules and related
their use to the error backpropagation method for training multilayer ANNs.
Williams (1988) described several ways that backpropagation and reinforcement
learning can be combined for training ANNs. Williams (1992) showed that a
special case of the AR P algorithm is a REINFORCE algorithm, although better
results were obtained with the general AR P algorithm (Barto, 1985).

15.13.

Summary419

The third phase of interest in teams of reinforcement learning agents was influenced by increased understanding of the role of dopamine as a widely broadcast
neuromodulator and speculation about the existence of reward-modulated STDP.
Much more so than earlier research, this research considers details of synaptic
plasticity and other constraints from neuroscience. Publications include thefollowing (chronologically and alphabetically): Bartlett and Baxter (1999, 2000),
Xie and Seung (2004), Baras and Meir (2007), Farries and Fairhall (2007), Florian(2007), Izhikevich (2007), Pecevski, Maass, and Legenstein (2008), Legenstein,
Pecevski, and Maass (2008), Kolodziejski, Porr, and Wörgötter (2009), Urbanczik
and Senn (2009), and Vasilaki, Frémaux, Urbanczik, Senn, and Gerstner (2009).
Nowé, Vrancx, and De Hauwere (2012) reviewed more recent developments inthe wider field of multi-agent reinforcement learning
15.11 Yin and Knowlton (2006) reviewed findings from outcome-devaluation experiments with rodents supporting the view that habitual and goal-directed behavior(as psychologists use the phrase) are respectively most associated with processing in the dorsolateral striatum (DLS) and the dorsomedial striatum (DMS).
Results of functional imaging experiments with human subjects in the outcomedevaluation setting by Valentin, Dickinson, and O’Doherty (2007) suggest thatthe orbitofrontal cortex (OFC) is an important component of goal-directed choice.
Single unit recordings in monkeys by Padoa-Schioppa and Assad (2006) support
the role of the OFC in encoding values guiding choice behavior. Rangel, Camerer,and Montague (2008) and Rangel and Hare (2010) reviewed findings from theperspective of neuroeconomics about how the brain makes goal-directed decisions.
Pezzulo, van der Meer, Lansink, and Pennartz (2014) reviewed the neuroscienceof internally generated sequences and presented a model of how these mechanisms might be components of model-based planning. Daw and Shohamy (2008)
proposed that while dopamine signaling connects well to habitual, or model-free,behavior, other processes are involved in goal-directed, or model-based, behavior.
Data from experiments by Bromberg-Martin, Matsumoto, Hong, and Hikosaka(2010) indicate that dopamine signals contain information pertinent to both
habitual and goal-directed behavior. Doll, Simon, and Daw (2012) argued that
there may not a clear separation in the brain between mechanisms that subserve
habitual and goal-directed learning and choice.
15.12 Keiflin and Janak (2015) reviewed connections between TD errors and addiction.
Nutt, Lingford-Hughes, Erritzoe, and Stokes (2015) critically evaluated the
hypothesis that addiction is due to a disorder of the dopamine system. Montague,
Dolan, Friston, and Dayan (2012) outlined the goals and early e↵orts in the fieldof computational psychiatry, and Adams, Huys, and Roiser (2015) reviewed more
recent progress.

Chapter 16

Applications and Case Studies
In this chapter we present a few case studies of reinforcement learning. Several of theseare substantial applications of potential economic significance. One, Samuel’s checkers
player, is primarily of historical interest. Our presentations are intended to illustrate someof the trade-o↵s and issues that arise in real applications. For example, we emphasize how
domain knowledge is incorporated into the formulation and solution of the problem. We
also highlight the representation issues that are so often critical to successful applications.
The algorithms used in some of these case studies are substantially more complex thanthose we have presented in the rest of the book. Applications of reinforcement learning
are still far from routine and typically require as much art as science. Making applications
easier and more straightforward is one of the goals of current research in reinforcement
learning.

16.1

TD-Gammon

One of the most impressive applications of reinforcement learning to date is that by
Gerald Tesauro to the game of backgammon (Tesauro, 1992, 1994, 1995, 2002). Tesauro’s
program, TD-Gammon, required little backgammon knowledge, yet learned to playextremely well, near the level of the world’s strongest grandmasters. The learningalgorithm in TD-Gammon was a straightforward combination of the TD( ) algorithm
and nonlinear function approximation using a multilayer artificial neural network (ANN)trained by backpropagating TD errors.
Backgammon is a major game in the sense that it is played throughout the world,with numerous tournaments and regular world championship matches. It is in part agame of chance, and it is a popular vehicle for waging significant sums of money. There
are probably more professional backgammon players than there are professional chessplayers. The game is played with 15 white and 15 black pieces on a board of 24 locations,called points. To the right on the next page is shown a typical position early in the game,seen from the perspective of the white player. White here has just rolled the dice andobtained a 5 and a 2. This means that he can move one of his pieces 5 steps and one

422

Chapter 16: Applications and Case Studies(possibly the same piece) 2 steps. For
white pieces move
example, he could move two pieces from24 23 22 21 20 19
18 17 16 15 14 13
counterclockwise
the 12 point, one to the 17 point, andone to the 14 point. White’s objective isto advance all of his pieces into the lastquadrant (points 19–24) and then o↵
the board. The first player to remove
all his pieces wins. One complicationis that the pieces interact as they pass
black pieces
1 2 3 4 5 6
7 8 9 10 11 12each other going in di↵erent directions.
move clockwise
For example, if it were black’s move, he
could use the dice roll of 2 to move a
A backgammon position
piece from the 24 point to the 22 point,“hitting” the white piece there. Pieces that have been hit are placed on the “bar” in the
middle of the board (where we already see one previously hit black piece), from whence
they reenter the race from the start. However, if there are two pieces on a point, thenthe opponent cannot move to that point; the pieces are protected from being hit. Thus,
white cannot use his 5–2 dice roll to move either of his pieces on the 1 point, because
their possible resulting points are occupied by groups of black pieces. Forming contiguous
blocks of occupied points to block the opponent is one of the elementary strategies of thegame.
Backgammon involves several further complications, but the above description gives
the basic idea. With 30 pieces and 24 possible locations (26, counting the bar ando↵-the-board) it should be clear that the number of possible backgammon positions isenormous, far more than the number of memory elements one could have in any physicallyrealizable computer. The number of moves possible from each position is also large. For atypical dice roll there might be 20 di↵erent ways of playing. In considering future moves,such as the response of the opponent, one must consider the possible dice rolls as well.
The result is that the game tree has an e↵ective branching factor of about 400. This is
far too large to permit e↵ective use of the conventional heuristic search methods thathave proved so e↵ective in games like chess and checkers.
On the other hand, the game is a good match to the capabilities of TD learning
methods. Although the game is highly stochastic, a complete description of the game’s
state is available at all times. The game evolves over a sequence of moves and positions
until finally ending in a win for one player or the other, ending the game. The outcome
can be interpreted as a final reward to be predicted. On the other hand, the theoreticalresults we have described so far cannot be usefully applied to this task. The number ofstates is so large that a lookup table cannot be used, and the opponent is a source of
uncertainty and time variation.
TD-Gammon used a nonlinear form of TD( ). The estimated value, v̂(s,w), of any
state (board position) s was meant to estimate the probability of winning starting fromstate s. To achieve this, rewards were defined as zero for all time steps except thoseon which the game is won. To implement the value function, TD-Gammon used astandard multilayer ANN, much like that shown to the right on the next page. (The real

16.1. TD-Gammon

general update rule for this case iswhere wt is the vector ofiall modifiable parameters (in this case, the weighhof theof eligibility
traces, one for each componen
wt+1 = wt + Rt+1 +v̂(Snetwork)v̂(Ste,w(15.1)t is
t+1 ,wt ) andt ) aevectort,of wt , updated bywhere wt is the vector of all modifiable parameters (in this case, the weights
423
et 1 + traces,
rwt v̂(Sone), each component
t ,wtforof the network) and et is a vectoretof=eligibilityof wt , updated bywith e = 0. The gradient in this equation can be computed efficiently by th0

backpropagation procedure. For the backgammon application, in which =et = units
et 1 +
wt v̂(St ,wt ),network had two additional
inritspredicted
and the reward is always
zero exceptprobability
upon winning, the TD error portion of thrule is usually
,w) V
v̂(S
,w),
ofv̂(S
winning,t+1efficiently
final layer to estimatewith
thee0probabilityof learning= 0. The gradientin this equation
can be justcomputedtheas suggested in Figure 15.
t tby
backpropagation
backgammon
1
Tothe
apply
the learningapplication,
rule we needinawhichsource of=backgammon
games. Tesaur
each player’s winningin a specialprocedure.
way Forand the reward is always zero
exceptupon
winning,sequence
the TD error
portionof the his learning backgammo
TDan
error
obtained
unending
of
games
by
playing
called a “gammon” or“backgammon.”)TD
error,
V!
V
learning rule is usually justplayerv̂(St+1against,w) t+1v̂(Ssuggested
in Figure
15.2.
tToaschooset ,w),
itself.
its moves,
TD-Gammon
considered each of th
The network consisted of a layer of input 20 or so ways it could play its dice roll and the corresponding positions tha
To apply the learning rule we need a source of backgammon
games.
Tesauro. . . . . .afterstates as discussed in Section 6.
wouldofresult.
resultingunits, a layer of hidden
units,ananda finalobtained
unending
sequence
gamesTheby playing
hispositions
learningare
backgammon
The
network
was
consulted
toestimateeach
eachofofthevalues. The move wa
playerto
againstitself. To choose its moves, TD-Gammon considered
output unit. The input
the network
hidden. . .withtheir.
.
.
hiddenunits
units (40-8thenselected
that thewouldlead to the position
the highest
estimated
valu
20
or
so
ways
it
could
play
its
dice
roll
and
corresponding
positionsthat
was a representation of a backgammon Continuing in this way, with TD-Gammon making the(40-80)
moves for both side
would result. The resulting positions are afterstates as discussed in Section 6.8.
was possible to easily generate large numbers of backgammon games. Eac
position, and the output
an wasestimate
The wasnetwork
consultedit toestimate each of their values. The move wasgame was treated as an episode, with the sequence of positions acting aof the value of that position.
then selected that would lead to the position with the highest estimated value.
. nonlinear TD rule (15.1) ful. . applied. . . thethe states, S0 , S1 , S2 , . . .. TesauroContinuingin this way, with TD-Gammon making the moves for both sides,
In the first versionof TD-Gammon,incrementally, that is, after each individual move.
backgammon
position(198
input units)
it was possible to easily generate large numbers of backgammon games. Each
TD-Gammon 0.0, backgammon
The weights
the network
were set initiallyto small random values. Th
game was treated posias an episode,with theof sequenceof positions
acting as
initial evaluations
thus entirely
arbitrary.
Since the moves were selectetions were represented
to thethe states,
S0 ,network
S1 , S2 , . . .. Tesauro
applied thewere
nonlinearTD rule(15.1) fully
Figure
16.1:
The TD-GammonANNtheindividualbasisof these
evaluations,the initial moves
were inevitably poor, anthat is, afteroneachmove.
in a relatively directincrementally,
way that involved
the initial games often lasted hundreds or thousands of moves before one sid
The weights of
network
were
set initiallyto smalllittle backgammon knowledge.
Itthe
did,
or theotherwon, almost
by random
accident.values.
AfterThea few dozen games howeve
initial evaluations
were thus
entirely
arbitrary.
Since
the movesselected
performance
improved
rapidly.
however, involve substantial
knowledgeofhow ANNs
work
and
how were
information
is beston the basis of these evaluations, the initial moves were inevitably poor, andpresented to them. Ittheis initial
instructive
to
note
the
exact
representation
Tesaurochose.
There
After
playing
about
300,000
games
againstitself,
TD-Gammon
0.0 as dgames often lasted hundreds or thousands of moves before one side
scribedaboveeachlearnedto play
approximately
as wellboard,as the best previou
were a total of 198 input
thealmost
network.
For
point
on
the
backgammonor theunits
other towon,by accident.
After
a few
dozen
games
however,backgammon computer programs. This was a striking result because all th
improved
rapidly.
four units indicated performancethe numberof whitepieces on the point. If there were no white
previous high-performance computer programs had used extensive backgam
Aftertookplayinggames
TD-Gammon
0.0 asde- the program
pieces, then all four units
onabout
the 300,000
value
zero. against
If there
was
one
then
first at the tim
mon knowledge.
Foritself,
example,the piece,
reigning
champion
scribedabove encoded
learned to theplay elementary
approximately
as well asanotherprevious
was,arguably,
Neurogammon,
programwrittenby Tesauro that use
unit took on the value
1. Thisconcept
ofthea best
“blot,”
i.e.,
a piecebackgammon computer programs.
wasbuta striking
becauseall thea neural This
network
not TDresult
learning.
Neurogammon’s
network was trainethat can be hit by the
opponent.
If thereon
were
two
or morepieces,thenmoves
the second
unit
previous
high-performancecomputerprograms
had used
extensive
backgama large training
corpusof exemplary
provided by backgammon ex
was set to 1. This encoded
the basic
conceptof reigning
a in“madepoint”
onwhich
opponent
mon knowledge.
For example,
championprogramatthetheperts,theand,addition,startedwitha set
oftimefeaturesspecially crafted fo
arguably,
Neurogammon,
another
writtenby Tesaurothat usedcannot land. If therewas,
wereexactly
three pieces
on program
the point,then
the third
unit was set
a neural network but not TD learning. Neurogammon’s network was trained
to 1. This encoded thebasic
conceptofa“singlespare,”
i.e.,
an
extra
piece
in addition
on a large training corpus of exemplary moves provided by backgammon exto the two pieces thatmadepoint. Finally,
if theremorethancrafted
threeforpieces, theperts,
and,thein addition,
started witha set ofwerefeaturesspecially

fourth unit was set to a value proportionate to the number of additional pieces beyond
three. Letting n denote the total number of pieces on the point, if n > 3, then the fourth
unit took on the value (n 3)/2. This encoded a linear representation of “multiple spares”
at the given point.
With four units for white and four for black at each of the 24 points, that made a
total of 192 units. Two additional units encoded the number of white and black pieces on
the bar (each took the value n/2, where n is the number of pieces on the bar), and two
more encoded the number of black and white pieces already successfully removed fromthe board (these took the value n/15, where n is the number of pieces already borneo↵). Finally, two units indicated in a binary fashion whether it was white’s or black’s
turn to move. The general logic behind these choices should be clear. Basically, Tesauro
tried to represent the position in a straightforward way, while keeping the number ofunits relatively small. He provided one unit for each conceptually distinct possibility that
seemed likely to be relevant, and he scaled them to roughly the same range, in this casebetween 0 and 1.
Given a representation of a backgammon position, the network computed its estimatedvalue in the standard way. Corresponding to each connection from an input unit to ahidden unit was a real-valued weight. Signals from each input unit were multiplied by424

Chapter 16: Applications and Case Studies

their corresponding weights and summed at the hidden unit. The output, h(j), of hidden
unit j was a nonlinear sigmoid function of the weighted sum:!
X1Ph(j) =
wij xi =,i wij xi
1+eiwhere xi is the value of the ith input unit and wij is the weight of its connection to thejth hidden unit (all the weights in the network together make up the parameter vector w).
The output of the sigmoid is always between 0 and 1, and has a natural interpretation asa probability based on a summation of evidence. The computation from hidden units
to the output unit was entirely analogous. Each connection from a hidden unit to the
output unit had a separate weight. The output unit formed the weighted sum and thenpassed it through the same sigmoid nonlinearity.
TD-Gammon used the semi-gradient form of the TD( ) algorithm described in Section 12.2, with the gradients computed by the error backpropagation algorithm (Rumelhart, Hinton, and Williams, 1986). Recall that the general update rule for this case ishi.
wt+1 = wt + ↵ Rt+1 + v̂(St+1 ,wt ) v̂(St ,wt ) zt ,(16.1)

where wt is the vector of all modifiable parameters (in this case, the weights of the
network) and zt is a vector of eligibility traces, one for each component of wt , updated by.
zt = zt 1 + rv̂(St ,wt ),.
with z0 = 0. The gradient in this equation can be computed efficiently by the backpropagation procedure. For the backgammon application, in which = 1 and the reward isalways zero except upon winning, the TD error portion of the learning rule is usuallyjust v̂(St+1 ,w) v̂(St ,w), as suggested in Figure 16.1.
To apply the learning rule we need a source of backgammon games. Tesauro obtained
an unending sequence of games by playing his learning backgammon player against itself.
To choose its moves, TD-Gammon considered each of the 20 or so ways it could playits dice roll and the corresponding positions that would result. The resulting positions
are afterstates as discussed in Section 6.8. The network was consulted to estimate eachof their values. The move was then selected that would lead to the position with thehighest estimated value. Continuing in this way, with TD-Gammon making the moves
for both sides, it was possible to easily generate large numbers of backgammon games.
Each game was treated as an episode, with the sequence of positions acting as the states,
S0 , S1 , S2 , . . .. Tesauro applied the nonlinear TD rule (16.1) fully incrementally, that is,after each individual move.
The weights of the network were set initially to small random values. The initial
evaluations were thus entirely arbitrary. Because the moves were selected on the basisof these evaluations, the initial moves were inevitably poor, and the initial games often
lasted hundreds or thousands of moves before one side or the other won, almost byaccident. After a few dozen games however, performance improved rapidly.
After playing about 300,000 games against itself, TD-Gammon 0.0 as described above
learned to play approximately as well as the best previous backgammon computer16.1. TD-Gammon

425

programs. This was a striking result because all the previous high-performance computer
programs had used extensive backgammon knowledge. For example, the reigning champion
program at the time was, arguably, Neurogammon, another program written by Tesaurothat used an ANN but not TD learning. Neurogammon’s network was trained on alarge training corpus of exemplary moves provided by backgammon experts, and, in
addition, started with a set of features specially crafted for backgammon. Neurogammon
was a highly tuned, highly e↵ective backgammon program that decisively won theWorld Backgammon Olympiad in 1989. TD-Gammon 0.0, on the other hand, wasconstructed with essentially zero backgammon knowledge. That it was able to do as
well as Neurogammon and all other approaches is striking testimony to the potential ofself-play learning methods.
The tournament success of TD-Gammon 0.0 with zero expert backgammon knowledge
suggested an obvious modification: add the specialized backgammon features but keep
the self-play TD learning method. This produced TD-Gammon 1.0. TD-Gammon 1.0 was
clearly substantially better than all previous backgammon programs and found serious
competition only among human experts. Later versions of the program, TD-Gammon 2.0(40 hidden units) and TD-Gammon 2.1 (80 hidden units), were augmented with a selective
two-ply search procedure. To select moves, these programs looked ahead not just to thepositions that would immediately result, but also to the opponent’s possible dice rolls
and moves. Assuming the opponent always took the move that appeared immediately
best for him, the expected value of each candidate move was computed and the best
was selected. To save computer time, the second ply of search was conducted only forcandidate moves that were ranked highly after the first ply, about four or five moves on
average. Two-ply search a↵ected only the moves selected; the learning process proceeded
exactly as before. The final versions of the program, TD-Gammon 3.0 and 3.1, used 160
hidden units and a selective three-ply search. TD-Gammon illustrates the combinationof learned value functions and decision-time search as in heuristic search and MCTS
methods. In follow-on work, Tesauro and Galperin (1997) explored trajectory sampling
methods as an alternative to full-width search, which reduced the error rate of live playby large numerical factors (4x–6x) while keeping the think time reasonable at ⇠5–10
seconds per move.
During the 1990s, Tesauro was able to play his programs in a significant number ofgames against world-class human players. A summary of the results is given in Table 16.1.

Program
TD-Gammon 0.0
TD-Gammon 1.0
TD-Gammon 2.0
TD-Gammon 2.1
TD-Gammon 3.0

Hidden
Units
40
80
40
80
80

Training
Games
300,000
300,000
800,000
1,500,000
1,500,000

Opponents

Resultsother programs
Robertie, Magriel, ...
various Grandmasters
Robertie
Kazarostied for best
13 pts / 51 games
7 pts / 38 games
1 pt / 40 games+6 pts / 20 games

Table 16.1: Summary of TD-Gammon Results

426

Chapter 16: Applications and Case Studies

Based on these results and analyses by backgammon grandmasters (Robertie, 1992; seeTesauro, 1995), TD-Gammon 3.0 appeared to play at close to, or possibly better than,the playing strength of the best human players in the world. Tesauro reported in a
subsequent article (Tesauro, 2002) the results of an extensive rollout analysis of the movedecisions and doubling decisions of TD-Gammon relative to top human players. The
conclusion was that TD-Gammon 3.1 had a “lopsided advantage” in piece-movement
decisions, and a “slight edge” in doubling decisions, over top humans.
TD-Gammon had a significant impact on the way the best human players play thegame. For example, it learned to play certain opening positions di↵erently than was
the convention among the best human players. Based on TD-Gammon’s success andfurther analysis, the best human players now play these positions as TD-Gammon does
(Tesauro, 1995). The impact on human play was greatly accelerated when several otherself-teaching ANN backgammon programs inspired by TD-Gammon, such as Jellyfish,
Snowie, and GNUBackgammon, became widely available. These programs enabled wide
dissemination of new knowledge generated by the ANNs, resulting in great improvementsin the overall caliber of human tournament play (Tesauro, 2002).

16.2

Samuel’s Checkers Player

An important precursor to Tesauro’s TD-Gammon was the seminal work of Arthur Samuel(1959, 1967) in constructing programs for learning to play checkers. Samuel was one of
the first to make e↵ective use of heuristic search methods and of what we would now call
temporal-di↵erence learning. His checkers players are instructive case studies in addition
to being of historical interest. We emphasize the relationship of Samuel’s methods tomodern reinforcement learning methods and try to convey some of Samuel’s motivation
for using them.
Samuel first wrote a checkers-playing program for the IBM 701 in 1952. His firstlearning program was completed in 1955 and was demonstrated on television in 1956.
Later versions of the program achieved good, though not expert, playing skill. Samuelwas attracted to game-playing as a domain for studying machine learning because games
are less complicated than problems “taken from life” while still allowing fruitful study of
how heuristic procedures and learning can be used together. He chose to study checkers
instead of chess because its relative simplicity made it possible to focus more strongly on
learning.
Samuel’s programs played by performing a lookahead search from each current position.
They used what we now call heuristic search methods to determine how to expand thesearch tree and when to stop searching. The terminal board positions of each search were
evaluated, or “scored,” by a value function, or “scoring polynomial,” using linear function
approximation. In this and other respects Samuel’s work seems to have been inspiredby the suggestions of Shannon (1950). In particular, Samuel’s program was based on
Shannon’s minimax procedure to find the best move from the current position. Working
backward through the search tree from the scored terminal positions, each position wasgiven the score of the position that would result from the best move, assuming that themachine would always try to maximize the score, while the opponent would always try to16.2. Samuel’s Checkers Player

427

minimize it. Samuel called this the “backed-up score” of the position. When the minimax
procedure reached the search tree’s root—the current position—it yielded the best move
under the assumption that the opponent would be using the same evaluation criterion,
shifted to its point of view. Some versions of Samuel’s programs used sophisticated searchcontrol methods analogous to what are known as “alpha-beta” cuto↵s (e.g., see Pearl,1984).
Samuel used two main learning methods, the simplest of which he called rote learning.
It consisted simply of saving a description of each board position encountered during play
together with its backed-up value determined by the minimax procedure. The result wasthat if a position that had already been encountered were to occur again as a terminal
position of a search tree, the depth of the search was e↵ectively amplified because thisposition’s stored value cached the results of one or more searches conducted earlier. One
initial problem was that the program was not encouraged to move along the most direct
path to a win. Samuel gave it a “a sense of direction” by decreasing a position’s value
a small amount each time it was backed up a level (called a ply) during the minimax
analysis. “If the program is now faced with a choice of board positions whose scoresdi↵er only by the ply number, it will automatically make the most advantageous choice,choosing a low-ply alternative if winning and a high-ply alternative if losing” (Samuel,1959, p. 80). Samuel found this discounting-like technique essential to successful learning.
Rote learning produced slow but continual improvement that was most e↵ective for
opening and endgame play. His program became a “better-than-average novice” after
learning from many games against itself, a variety of human opponents, and from bookgames in a supervised learning mode.
Rote learning and other aspects of Samuel’s work strongly suggest the essential ideaof temporal-di↵erence learning—that the value of a state should equal the value of
likely following states. Samuel came closest to this idea in his second learning method,
his “learning by generalization” procedure for modifying the parameters of the valuefunction. Samuel’s method was the same in concept as that used much later by Tesauro
in TD-Gammon. He played his program many games against another version of itself andperformed an update after each move. The idea of Samuel’s update is suggested by the
backup diagram in Figure 16.2. Each open circle represents a position where the program
moves next, an on-move position, and each solid circle represents a position where theopponent moves next. An update was made to the value of each on-move position after amove by each side, resulting in a second on-move position. The update was toward theminimax value of a search launched from the second on-move position. Thus, the overalle↵ect was that of a backing-up over one full move of real events and then a search overpossible events, as suggested by Figure 16.2. Samuel’s actual algorithm was significantly
more complex than this for computational reasons, but this was the basic idea.
Samuel did not include explicit rewards. Instead, he fixed the weight of the mostimportant feature, the piece advantage feature, which measured the number of pieces
the program had relative to how many its opponent had, giving higher weight to kings,
and including refinements so that it was better to trade pieces when winning than whenlosing. Thus, the goal of Samuel’s program was to improve its piece advantage, which in
checkers is highly correlated with winning.

428

Chapter 16: Applications and Case Studies

actual events
hypothetical events

backupFigure 16.2: The backup diagram for Samuel’s checkers player.

However, Samuel’s learning method may have been missing an essential part of a soundtemporal-di↵erence algorithm. Temporal-di↵erence learning can be viewed as a way ofmaking a value function consistent with itself, and this we can clearly see in Samuel’s
method. But also needed is a way of tying the value function to the true value of thestates. We have enforced this via rewards and by discounting or giving a fixed value tothe terminal state. But Samuel’s method included no rewards and no special treatment of
the terminal positions of games. As Samuel himself pointed out, his value function could
have become consistent merely by giving a constant value to all positions. He hoped to
discourage such solutions by giving his piece-advantage term a large, nonmodifiable weight.
But although this may decrease the likelihood of finding useless evaluation functions,
it does not prohibit them. For example, a constant function could still be attained by
setting the modifiable weights so as to cancel the e↵ect of the nonmodifiable one.
Because Samuel’s learning procedure was not constrained to find useful evaluation
functions, it should have been possible for it to become worse with experience. In fact,
Samuel reported observing this during extensive self-play training sessions. To get the
program improving again, Samuel had to intervene and set the weight with the largest
absolute value back to zero. His interpretation was that this drastic intervention jarred
the program out of local optima, but another possibility is that it jarred the program outof evaluation functions that were consistent but had little to do with winning or losing
the game.
Despite these potential problems, Samuel’s checkers player using the generalization
learning method approached “better-than-average” play. Fairly good amateur opponents
characterized it as “tricky but beatable” (Samuel, 1959). In contrast to the rote-learning
version, this version was able to develop a good middle game but remained weak in
opening and endgame play. This program also included an ability to search through sets offeatures to find those that were most useful in forming the value function. A later version(Samuel, 1967) included refinements in its search procedure, such as alpha-beta pruning,16.3. Watson’s Daily-Double Wagering

429

extensive use of a supervised learning mode called “book learning,” and hierarchicallookup tables called signature tables (Griffith, 1966) to represent the value function
instead of linear function approximation. This version learned to play much better than
the 1959 program, though still not at a master level. Samuel’s checkers-playing program
was widely recognized as a significant achievement in artificial intelligence and machinelearning.

16.3

Watson’s Daily-Double Wagering

IBM Watson1 is the system developed by a team of IBM researchers to play the popular
TV quiz show Jeopardy!.2 It gained fame in 2011 by winning first prize in an exhibition
match against human champions. Although the main technical achievement demonstrated
by Watson was its ability to quickly and accurately answer natural language questions
over broad areas of general knowledge, its winning Jeopardy! performance also relied on
sophisticated decision-making strategies for critical parts of the game. Tesauro, Gondek,
Lechner, Fan, and Prager (2012, 2013) adapted Tesauro’s TD-Gammon system describedabove to create the strategy used by Watson in “Daily-Double” (DD) wagering in its
celebrated winning performance against human champions. These authors report that thee↵ectiveness of this wagering strategy went well beyond what human players are able to
do in live game play, and that it, along with other advanced strategies, was an important
contributor to Watson’s impressive winning performance. Here we focus only on DD
wagering because it is the component of Watson that owes the most to reinforcement
learning.
Jeopardy! is played by three contestants who face a board showing 30 squares, each ofwhich hides a clue and has a dollar value. The squares are arranged in six columns, eachcorresponding to a di↵erent category. A contestant selects a square, the host reads thesquare’s clue, and each contestant may choose to respond to the clue by sounding a buzzer(“buzzing in”). The first contestant to buzz in gets to try responding to the clue. If thiscontestant’s response is correct, their score increases by the dollar value of the square; if
their response is not correct, or if they do not respond within five seconds, their score
decreases by that amount, and the other contestants get a chance to buzz in to respondto the same clue. One or two squares (depending on the game’s current round) are
special DD squares. A contestant who selects one of these gets an exclusive opportunity
to respond to the square’s clue and has to decide—before the clue is revealed—on how
much to wager, or bet. The bet has to be greater than five dollars but not greater thanthe contestant’s current score. If the contestant responds correctly to the DD clue, theirscore increases by the bet amount; otherwise it decreases by the bet amount. At the endof each game is a “Final Jeopardy” (FJ) round in which each contestant writes downa sealed bet and then writes an answer after the clue is read. The contestant with thehighest score after three rounds of play (where a round consists of revealing all 30 clues)
is the winner. The game has many other details, but these are enough to appreciate
1 Registered trademark of IBM Corp.
2 Registered trademark of Jeopardy Productions Inc.

430

Chapter 16: Applications and Case Studies

the importance of DD wagering. Winning or losing often depends on a contestant’s DD
wagering strategy.
Whenever Watson selected a DD square, it chose its bet by comparing action values,q̂(s, bet), that estimated the probability of a win from the current game state, s, foreach round-dollar legal bet. Except for some risk-abatement measures described below,
Watson selected the bet with the maximum action value. Action values were computed
whenever a betting decision was needed by using two types of estimates that were learned
before any live game play took place. The first were estimated values of the afterstates(Section 6.8) that would result from selecting each legal bet. These estimates were obtained
from a state-value function, v̂(·,w), defined by parameters w, that gave estimates of theprobability of a win for Watson from any game state. The second estimates used tocompute action values gave the “in-category DD confidence,” pDD , which estimated thelikelihood that Watson would respond correctly to the as-yet unrevealed DD clue.
Tesauro et al. used the reinforcement learning approach of TD-Gammon described above
to learn v̂(·,w): a straightforward combination of nonlinear TD( ) using a multilayer
ANN with weights w trained by backpropagating TD errors during many simulated
games. States were represented to the network by feature vectors specifically designed
for Jeopardy!. Features included the current scores of the three players, how many DDs
remained, the total dollar value of the remaining clues, and other information related tothe amount of play left in the game. Unlike TD-Gammon, which learned by self-play,
Watson’s v̂ was learned over millions of simulated games against carefully-crafted models
of human players. In-category confidence estimates were conditioned on the number ofright responses r and wrong responses w that Watson gave in previously-played clues in
the current category. The dependencies on (r, w) were estimated from Watson’s actual
accuracies over many thousands of historical categories.
With the previously learned value function v̂ and in-category DD confidence pDD ,Watson computed q̂(s, bet) for each legal round-dollar bet as follows:q̂(s, bet) = pDD ⇥ v̂(SW + bet,. . .) + (1pDD ) ⇥ v̂(SWbet,. . .),(16.2)

where SW is Watson’s current score, and v̂ gives the estimated value for the game state
after Watson’s response to the DD clue, which is either correct or incorrect. Computing
an action value this way corresponds to the insight from Exercise 3.19 that an action
value is the expected next state value given the action (except that here it is the expected
next afterstate value because the full next state of the entire game depends on the next
square selection).
Tesauro et al. found that selecting bets by maximizing action values incurred “afrightening amount of risk,” meaning that if Watson’s response to the clue happened tobe wrong, the loss could be disastrous for its chances of winning. To decrease the downsiderisk of a wrong answer, Tesauro et al. adjusted (16.2) by subtracting a small fraction of
the standard deviation over Watson’s correct/incorrect afterstate evaluations. They
further reduced risk by prohibiting bets that would cause the wrong-answer afterstatevalue to decrease below a certain limit. These measures slightly reduced Watson’s
expectation of winning, but they significantly reduced downside risk, not only in terms of
average risk per DD bet, but even more so in extreme-risk scenarios where a risk-neutral
Watson would bet most or all of its bankroll.

16.3. Watson’s Daily-Double Wagering

431

Why was the TD-Gammon method of self-play not used to learn the critical valuefunction v̂? Learning from self-play in Jeopardy! would not have worked very wellbecause Watson was so di↵erent from any human contestant. Self-play would have led to
exploration of state space regions that are not typical for play against human opponents,
particularly human champions. In addition, unlike backgammon, Jeopardy! is a gameof imperfect information because contestants do not have access to all the informationinfluencing their opponents’ play. In particular, Jeopardy! contestants do not know how
much confidence their opponents have for responding to clues in the various categories.
Self-play would have been something like playing poker with someone who is holding thesame cards that you hold.
As a result of these complications, much of the e↵ort in developing Watson’s DDwagering strategy was devoted to creating good models of human opponents. The models
did not address the natural language aspect of the game, but were instead stochastic
process models of events that can occur during play. Statistics were extracted from anextensive fan-created archive of game information from the beginning of the show to thepresent day. The archive includes information such as the ordering of the clues, right andwrong contestant answers, DD locations, and DD and FJ bets for nearly 300,000 clues.
Three models were constructed: an Average Contestant model (based on all the data), aChampion model (based on statistics from games with the 100 best players), and a Grand
Champion model (based on statistics from games with the 10 best players). In addition
to serving as opponents during learning, the models were used to assess the benefits
produced by the learned DD-wagering strategy. Watson’s win rate in simulation when it
used a baseline heuristic DD-wagering strategy was 61%; when it used the learned values
and a default confidence value, its win rate increased to 64%; and with live in-category
confidence, it was 67%. Tesauro et al. regarded this as a significant improvement, giventhat the DD wagering was needed only about 1.5 to 2 times in each game.
Because Watson had only a few seconds to bet, as well as to select squares and decide
whether or not to buzz in, the computation time needed to make these decisions wasa critical factor. The ANN implementation of v̂ allowed DD bets to be made quickly
enough to meet the time constraints of live play. However, once games could be simulated
fast enough through improvements in the simulation software, near the end of a gameit was feasible to estimate the value of bets by averaging over many Monte-Carlo trials
in which the consequence of each bet was determined by simulating play to the game’send. Selecting endgame DD bets in live play based on Monte-Carlo trials instead of the
ANN significantly improved Watson’s performance because errors in value estimatesin endgames could seriously a↵ect its chances of winning. Making all the decisions via
Monte-Carlo trials might have led to better wagering decisions, but this was simply
impossible given the complexity of the game and the time constraints of live play.
Although its ability to quickly and accurately answer natural language questions
stands out as Watson’s major achievement, all of its sophisticated decision strategies
contributed to its impressive defeat of human champions. According to Tesauro et al.
(2012):... it is plainly evident that our strategy algorithms achieve a level of quantitative precision and real-time performance that exceeds human capabilities.

432

Chapter 16: Applications and Case Studies

This is particularly true in the cases of DD wagering and endgame buzzing,
where humans simply cannot come close to matching the precise equity andconfidence estimates and complex decision calculations performed by Watson.

16.4

Optimizing Memory Control

Most computers use dynamic random access memory (DRAM) as their main memory
because of its low cost and high capacity. The job of a DRAM memory controller is toefficiently use the interface between the processor chip and an o↵-chip DRAM system
to provide the high-bandwidth and low-latency data transfer necessary for high-speed
program execution. A memory controller needs to deal with dynamically changing
patterns of read/write requests while adhering to a large number of timing and resource
constraints required by the hardware. This is a formidable scheduling problem, especiallywith modern processors with multiple cores sharing the same DRAM.
İpek, Mutlu, Martı́nez, and Caruana (2008) (also Martı́nez and İpek, 2009) designed
a reinforcement learning memory controller and demonstrated that it can significantly
improve the speed of program execution over what was possible with conventional
controllers at the time of their research. They were motivated by limitations of existingstate-of-the-art controllers that used policies that did not take advantage of past schedulingexperience and did not account for long-term consequences of scheduling decisions. İpek
et al.’s project was carried out by means of simulation, but they designed the controller
at the detailed level of the hardware needed to implement it—including the learningalgorithm—directly on a processor chip.
Accessing DRAM involves a number of steps that have to be done according to stricttime constraints. DRAM systems consist of multiple DRAM chips, each containing
multiple rectangular arrays of storage cells arranged in rows and columns. Each cell
stores a bit as the charge on a capacitor. Because the charge decreases over time, each
DRAM cell needs to be recharged—refreshed—every few milliseconds to prevent memory
content from being lost. This need to refresh the cells is why DRAM is called “dynamic.”
Each cell array has a row bu↵er that holds a row of bits that can be transferred intoor out of one of the array’s rows. An activate command “opens a row,” which means
moving the contents of the row whose address is indicated by the command into therow bu↵er. With a row open, the controller can issue read and write commands to thecell array. Each read command transfers a word (a short sequence of consecutive bits)in the row bu↵er to the external data bus, and each write command transfers a wordin the external data bus to the row bu↵er. Before a di↵erent row can be opened, aprecharge command must be issued which transfers the (possibly updated) data in therow bu↵er back into the addressed row of the cell array. After this, another activatecommand can open a new row to be accessed. Read and write commands are column
commands because they sequentially transfer bits into or out of columns of the row bu↵er;
multiple bits can be transferred without re-opening the row. Read and write commands
to the currently-open row can be carried out more quickly than accessing a di↵erent row,which would involve additional row commands: precharge and activate; this is sometimes

16.4. Optimizing Memory Control

433

referred to as “row locality.” A memory controller maintains a memory transaction queuethat stores memory-access requests from the processors sharing the memory system. The
controller has to process requests by issuing commands to the memory system while
adhering to a large number of timing constraints.
A controller’s policy for scheduling access requests can have a large e↵ect on theperformance of the memory system, such as the average latency with which requests
can be satisfied and the throughput the system is capable of achieving. The simplest
scheduling strategy handles access requests in the order in which they arrive by issuing
all the commands required by the request before beginning to service the next one. But if
the system is not ready for one of these commands, or executing a command would result
in resources being underutilized (e.g., due to timing constraints arising from servicingthat one command), it makes sense to begin servicing a newer request before finishing
the older one. Policies can gain efficiency by reordering requests, for example, by giving
priority to read requests over write requests, or by giving priority to read/write commands
to already open rows. The policy called First-Ready, First-Come-First-Serve (FR-FCFS),
gives priority to column commands (read and write) over row commands (activate andprecharge), and in case of a tie gives priority to the oldest command. FR-FCFS wasshown to outperform other scheduling policies in terms of average memory-access latency
under conditions commonly encountered (Rixner, 2004).
Figure 16.3 is a high-level view of İpek et al.’s reinforcement learning memory controller.
They modeled the DRAM access process as an MDP whose states are the contents of thetransaction queue and whose actions are commands to the DRAM system: precharge,
activate, read, write, and NoOp. The reward signal is 1 whenever the action is read or
write, and otherwise it is 0. State transitions were considered to be stochastic because
the next state of the system not only depends on the scheduler’s command, but also on
aspects of the system’s behavior that the scheduler cannot control, such as the workloadsof the processor cores accessing the DRAM system.

Figure 16.3: High-level view of the reinforcement learning DRAM controller. The scheduler isthe reinforcement learning agent. Its environment is represented by features of the transaction
queue, and its actions are commands to the DRAM system. ©2009 IEEE. Reprinted, withpermission, from J. F. Martı́nez and E. İpek, Dynamic multicore resource management: A
machine learning approach, Micro, IEEE, 29(5), p. 12.

434

Chapter 16: Applications and Case Studies

Critical to this MDP are constraints on the actions available in each state. Recall from
Chapter 3 that the set of available actions can depend on the state: At 2 A(St ), where
At is the action at time step t and A(St ) is the set of actions available in state St . In
this application, the integrity of the DRAM system was assured by not allowing actions
that would violate timing or resource constraints. Although İpek et al. did not make itexplicit, they e↵ectively accomplished this by pre-defining the sets A(St ) for all possible
states St .
These constraints explain why the MDP has a NoOp action and why the reward signal
is 0 except when a read or write command is issued. NoOp is issued when it is the sole
legal action in a state. To maximize utilization of the memory system, the controller’stask is to drive the system to states in which either a read or a write action can beselected: only these actions result in sending data over the external data bus, so it is only
these that contribute to the throughput of the system. Although precharge and activate
produce no immediate reward, the agent needs to select these actions to make it possible
to later select the rewarded read and write actions.
The scheduling agent used Sarsa (Section 6.4) to learn an action-value function. States
were represented by six integer-valued features. To approximate the action-value function,
the algorithm used linear function approximation implemented by tile coding with hashing(Section 9.5.4). The tile coding had 32 tilings, each storing 256 action values as 16-bit
fixed point numbers. Exploration was "-greedy with " = 0.05.
State features included the number of read requests in the transaction queue, thenumber of write requests in the transaction queue, the number of write requests in thetransaction queue waiting for their row to be opened, and the number of read requests inthe transaction queue waiting for their row to be opened that are the oldest issued bytheir requesting processors. (The other features depended on how the DRAM interacts
with cache memory, details we omit here.) The selection of the state features was basedon İpek et al.’s understanding of factors that impact DRAM performance. For example,
balancing the rate of servicing reads and writes based on how many of each are in thetransaction queue can help avoid stalling the DRAM system’s interaction with cachememory. The authors in fact generated a relatively long list of potential features, andthen pared them down to a handful using simulations guided by stepwise feature selection.
An interesting aspect of this formulation of the scheduling problem as an MDP isthat the features input to the tile coding for defining the action-value function weredi↵erent from the features used to specify the action-constraint sets A(St ). Whereas thetile coding input was derived from the contents of the transaction queue, the constraint
sets depended on a host of other features related to timing and resource constraints that
had to be satisfied by the hardware implementation of the entire system. In this way, theaction constraints ensured that the learning algorithm’s exploration could not endanger
the integrity of the physical system, while learning was e↵ectively limited to a “safe”
region of the much larger state space of the hardware implementation.
Because an objective of this work was that the learning controller could be implemented
on a chip so that learning could occur online while a computer is running, hardware
implementation details were important considerations. The design included two five-stage
pipelines to calculate and compare two action values at every processor clock cycle, and16.4. Optimizing Memory Control

435

to update the appropriate action value. This included accessing the tile coding which
was stored on-chip in static RAM. For the configuration İpek et al. simulated, which was
a 4GHz 4-core chip typical of high-end workstations at the time of their research, there
were 10 processor cycles for every DRAM cycle. Considering the cycles needed to fill
the pipes, up to 12 actions could be evaluated in each DRAM cycle. İpek et al. foundthat the number of legal commands for any state was rarely greater than this, and that
performance loss was negligible if enough time was not always available to consider all
legal commands. These and other clever design details made it feasible to implement thecomplete controller and learning algorithm on a multi-processor chip.
İpek et al. evaluated their learning controller in simulation by comparing it with threeother controllers: (1) the FR-FCFS controller mentioned above that produces the beston-average performance, (2) a conventional controller that processes each request inorder, and (3) an unrealizable ideal controller, called the Optimistic controller, able
to sustain 100% DRAM throughput if given enough demand by ignoring all timing
and resource constraints, but otherwise modeling DRAM latency (as row bu↵er hits)
and bandwidth. They simulated nine memory-intensive parallel workloads consisting of
scientific and data-mining applications. Figure 16.4 shows the performance (the inverseof execution time normalized to the performance of FR-FCFS) of each controller forthe nine applications, together with the geometric mean of their performances over theapplications. The learning controller, labeled RL in the figure, improved over that of
FR-FCFS by from 7% to 33% over the nine applications, with an average improvement of
19%. Of course, no realizable controller can match the performance of Optimistic, which
ignores all timing and resource constraints, but the learning controller’s performanceclosed the gap with Optimistic’s upper bound by an impressive 27%.
Because the rationale for on-chip implementation of the learning algorithm was toallow the scheduling policy to adapt online to changing workloads, İpek et al. analyzed
the impact of online learning compared to a previously-learned fixed policy. They trained

Figure 16.4: Performances of four controllers over a suite of 9 simulated benchmark applications.
The controllers are: the simplest ‘in-order’ controller, FR-FCFS, the learning controller RL,
and the unrealizable Optimistic controller which ignores all timing and resource constraints toprovide a performance upper bound. Performance, normalized to that of FR-FCFS, is the inverseof execution time. At far right is the geometric mean of performances over the 9 benchmark
applications for each controller. Controller RL comes closest to the ideal performance. ©2009
IEEE. Reprinted, with permission, from J. F. Martı́nez and E. İpek, Dynamic multicore resource
management: A machine learning approach, Micro, IEEE, 29(5), p. 13.

436

Chapter 16: Applications and Case Studies

their controller with data from all nine benchmark applications and then held the resulting
action values fixed throughout the simulated execution of the applications. They foundthat the average performance of the controller that learned online was 8% better thanthat of the controller using the fixed policy, leading them to conclude that online learning
is an important feature of their approach.
This learning memory controller was never committed to physical hardware because of
the large cost of fabrication. Nevertheless, İpek et al. could convincingly argue on the basisof their simulation results that a memory controller that learns online via reinforcement
learning has the potential to improve performance to levels that would otherwise require
more complex and more expensive memory systems, while removing from human designers
some of the burden required to manually design efficient scheduling policies. Mukundanand Martı́nez (2012) took this project forward by investigating learning controllers withadditional actions, other performance criteria, and more complex reward functions derived
using genetic algorithms. They considered additional performance criteria related toenergy efficiency. The results of these studies surpassed the earlier results describedabove and significantly surpassed the 2012 state-of-the-art for all of the performance
criteria they considered. The approach is especially promising for developing sophisticatedpower-aware DRAM interfaces.

16.5

Human-level Video Game PlayOne of the greatest challenges in applying reinforcement learning to real-world problems
is deciding how to represent and store value functions and/or policies. Unless the state
set is finite and small enough to allow exhaustive representation by a lookup table—as inmany of our illustrative examples—one must use a parameterized function approximation
scheme. Whether linear or nonlinear, function approximation relies on features thathave to be readily accessible to the learning system and able to convey the informationnecessary for skilled performance. Most successful applications of reinforcement learning
owe much to sets of features carefully handcrafted based on human knowledge andintuition about the specific problem to be tackled.
A team of researchers at Google DeepMind developed an impressive demonstration thata deep multi-layer ANN can automate the feature design process (Mnih et al., 2013, 2015).
Multi-layer ANNs have been used for function approximation in reinforcement learning
ever since the 1986 popularization of the backpropagation algorithm as a method forlearning internal representations (Rumelhart, Hinton, and Williams, 1986; see Section 9.7).
Striking results have been obtained by coupling reinforcement learning with backpropagation. The results obtained by Tesauro and colleagues with TD-Gammon and Watson
discussed above are notable examples. These and other applications benefited from theability of multi-layer ANNs to learn task-relevant features. However, in all the examplesof which we are aware, the most impressive demonstrations required the network’s input
to be represented in terms of specialized features handcrafted for the given problem.
This is vividly apparent in the TD-Gammon results. TD-Gammon 0.0, whose network
input was essentially a “raw” representation of the backgammon board, meaning that it
involved very little knowledge of backgammon, learned to play approximately as well as16.5. Human-level Video Game Play437

the best previous backgammon computer programs. Adding specialized backgammon
features produced TD-Gammon 1.0 which was substantially better than all previous
backgammon programs and competed well against human experts.
Mnih et al. developed a reinforcement learning agent called deep Q-network (DQN)
that combined Q-learning with a deep convolutional ANN, a many-layered, or deep,
ANN specialized for processing spatial arrays of data such as images. We describe deep
convolutional ANNs in Section 9.7. By the time of Mnih et al.’s work with DQN, deep
ANNs, including deep convolutional ANNs, had produced impressive results in many
applications, but they had not been widely used in reinforcement learning.
Mnih et al. used DQN to show how a reinforcement learning agent can achieve a highlevel of performance on any of a collection of di↵erent problems without having to usedi↵erent problem-specific feature sets. To demonstrate this, they let DQN learn to play
49 di↵erent Atari 2600 video games by interacting with a game emulator. DQN learned adi↵erent policy for each of the 49 games (because the weights of its ANN were reset torandom values before learning on each game), but it used the same raw input, networkarchitecture, and parameter values (e.g., step size, discount rate, exploration parameters,
and many more specific to the implementation) for all the games. DQN achieved levelsof play at or beyond human level on a large fraction of these games. Although the games
were alike in being played by watching streams of video images, they varied widely in otherrespects. Their actions had di↵erent e↵ects, they had di↵erent state-transition dynamics,
and they needed di↵erent policies for learning high scores. The deep convolutional ANN
learned to transform the raw input common to all the games into features specialized for
representing the action values required for playing at the high level DQN achieved formost of the games.
The Atari 2600 is a home video game console that was sold in various versions by AtariInc. from 1977 to 1992. It introduced or popularized many arcade video games that are
now considered classics, such as Pong, Breakout, Space Invaders, and Asteroids. Althoughmuch simpler than modern video games, Atari 2600 games are still entertaining andchallenging for human players, and they have been attractive as testbeds for developing
and evaluating reinforcement learning methods (Diuk, Cohen, Littman, 2008; Naddaf,
2010; Cobo, Zang, Isbell, and Thomaz, 2011; Bellemare, Veness, and Bowling, 2013).
Bellemare, Naddaf, Veness, and Bowling (2012) developed the publicly available Arcade
Learning Environment (ALE) to encourage and simplify using Atari 2600 games to study
learning and planning algorithms.
These previous studies and the availability of ALE made the Atari 2600 game collection
a good choice for Mnih et al.’s demonstration, which was also influenced by the impressive
human-level performance that TD-Gammon was able to achieve in backgammon. DQN
is similar to TD-Gammon in using a multi-layer ANN as the function approximation
method for a semi-gradient form of a TD algorithm, with the gradients computed by
the backpropagation algorithm. However, instead of using TD( ) as TD-Gammon did,DQN used the semi-gradient form of Q-learning. TD-Gammon estimated the values ofafterstates, which were easily obtained from the rules for making backgammon moves.
To use the same algorithm for the Atari games would have required generating the next
states for each possible action (which would not have been afterstates in that case).
This could have been done by using the game emulator to run single-step simulations

438

Chapter 16: Applications and Case Studies

for all the possible actions (which ALE makes possible). Or a model of each game’s
state-transition function could have been learned and used to predict next states (Oh,Guo, Lee, Lewis, and Singh, 2015). While these methods might have produced resultscomparable to DQN’s, they would have been more complicated to implement and would
have significantly increased the time needed for learning. Another motivation for using
Q-learning was that DQN used the experience replay method, described below, which
requires an o↵-policy algorithm. Being model-free and o↵-policy made Q-learning anatural choice.
Before describing the details of DQN and how the experiments were conducted, we lookat the skill levels DQN was able to achieve. Mnih et al. compared the scores of DQN withthe scores of the best performing learning system in the literature at the time, the scoresof a professional human games tester, and the scores of an agent that selected actions at
random. The best system from the literature used linear function approximation withfeatures designed using some knowledge about Atari 2600 games (Bellemare, Naddaf,
Veness, and Bowling, 2013). DQN learned on each game by interacting with the gameemulator for 50 million frames, which corresponds to about 38 days of experience withthe game. At the start of learning on each game, the weights of DQN’s network were resetto random values. To evaluate DQN’s skill level after learning, its score was averaged
over 30 sessions on each game, each lasting up to 5 minutes and beginning with a random
initial game state. The professional human tester played using the same emulator (withthe sound turned o↵ to remove any possible advantage over DQN which did not process
audio). After 2 hours of practice, the human played about 20 episodes of each game for up
to 5 minutes each and was not allowed to take any break during this time. DQN learned
to play better than the best previous reinforcement learning systems on all but 6 of thegames, and played better than the human player on 22 of the games. By considering any
performance that scored at or above 75% of the human score to be comparable to, or
better than, human-level play, Mnih et al. concluded that the levels of play DQN learned
reached or exceeded human level on 29 of the 46 games. See Mnih et al. (2015) for amore detailed account of these results.
For an artificial learning system to achieve these levels of play would be impressive
enough, but what makes these results remarkable—and what many at the time considered
to be breakthrough results for artificial intelligence—is that the very same learning systemachieved these levels of play on widely varying games without relying on any game-specificmodifications.
A human playing any of these 49 Atari games sees 210⇥160 pixel image frames with128 colors at 60Hz. In principle, exactly these images could have formed the raw input to
DQN, but to reduce memory and processing requirements, Mnih et al. preprocessed eachframe to produce an 84⇥84 array of luminance values. Because the full states of manyof the Atari games are not completely observable from the image frames, Mnih et al.
“stacked” the four most recent frames so that the inputs to the network had dimension
84⇥84⇥4. This did not eliminate partial observability for all of the games, but it was
helpful in making many of them more Markovian.
An essential point here is that these preprocessing steps were exactly the same for all 46
games. No game-specific prior knowledge was involved beyond the general understandingthat it should still be possible to learn good policies with this reduced dimension and16.5. Human-level Video Game Play

439

that stacking adjacent frames should help with the partial observability of some of thegames. Because no game-specific prior knowledge beyond this minimal amount was usedin preprocessing the image frames, we can think of the 84⇥84⇥4 input vectors as being“raw” input to DQN.
The basic architecture of DQN is similar to the deep convolutional ANN illustrated in
Figure 9.15 (though unlike that network, subsampling in DQN is treated as part of eachconvolutional layer, with feature maps consisting of units having only a selection of thepossible receptive fields). DQN has three hidden convolutional layers, followed by one
fully connected hidden layer, followed by the output layer. The three successive hidden
convolutional layers of DQN produce 32 20 ⇥ 20 feature maps, 64 9⇥ 9 feature maps,
and 64 7⇥7 feature maps. The activation function of the units of each feature map is a
rectifier nonlinearity (max(0, x)). The 3,136 (64⇥7⇥7) units in this third convolutional
layer all connect to each of 512 units in the fully connected hidden layer, which then eachconnect to all 18 units in the output layer, one for each possible action in an Atari game.
The activation levels of DQN’s output units were the estimated optimal action valuesof the corresponding state–action pairs, for the state represented by the network’s input.
The assignment of output units to a game’s actions varied from game to game, andbecause the number of valid actions varied between 4 and 18 for the games, not all output
units had functional roles in all of the games. It helps to think of the network as if itwere 18 separate networks, one for estimating the optimal action value of each possible
action. In reality, these networks shared their initial layers, but the output units learned
to use the features extracted by these layers in di↵erent ways.
DQN’s reward signal indicated how a games’s score changed from one time step tothe next: +1 whenever it increased, 1 whenever it decreased, and 0 otherwise. This
standardized the reward signal across the games and made a single step-size parameterwork well for all the games despite their varying ranges of scores. DQN used an "-greedy
policy, with " decreasing linearly over the first million frames and remaining at a low
value for the rest of the learning session. The values of the various other parameters,such as the learning step size, discount rate, and others specific to the implementation,
were selected by performing informal searches to see which values worked best for a small
selection of the games. These values were then held fixed for all of the games.
After DQN selected an action, the action was executed by the game emulator, which
returned a reward and the next video frame. The frame was preprocessed and added
to the four-frame stack that became the next input to the network. Skipping for the
moment the changes to the basic Q-learning procedure made by Mnih et al., DQN used
the following semi-gradient form of Q-learning to update the network’s weights:
hi
wt+1 = wt + ↵ Rt+1 + max q̂(St+1 , a, wt ) q̂(St , At , wt ) rq̂(St , At , wt ), (16.3)
awhere wt is the vector of the network’s weights, At is the action selected at time step t,
and St and St+1 are respectively the preprocessed image stacks input to the network attime steps t and t + 1.
The gradient in (16.3) was computed by backpropagation. Imagining again that there
was a separate network for each action, for the update at time step t, backpropagation
was applied only to the network corresponding to At . Mnih et al. took advantage oftechniques shown to improve the basic backpropagation algorithm when applied to large

440

Chapter 16: Applications and Case Studies

networks. They used a mini-batch method that updated weights only after accumulating
gradient information over a small batch of images (here after 32 images). This yieldedsmoother sample gradients compared to the usual procedure that updates weights after
each action. They also used a gradient-ascent algorithm called RMSProp (Tieleman and
Hinton, 2012) that accelerates learning by adjusting the step-size parameter for eachweight based on a running average of the magnitudes of recent gradients for that weight.
Mnih et al. modified the basic Q-learning procedure in three ways. First, they used
a method called experience replay first studied by Lin (1992). This method stores theagent’s experience at each time step in a replay memory that is accessed to perform theweight updates. It worked like this in DQN. After the game emulator executed action
At in a state represented by the image stack St , and returned reward Rt+1 and image
stack St+1 , it added the tuple (St , At , Rt+1 , St+1 ) to the replay memory. This memory
accumulated experiences over many plays of the same game. At each time step multiple Qlearning updates—a mini-batch—were performed based on experiences sampled uniformly
at random from the replay memory. Instead of St+1 becoming the new St for the next
update as it would in the usual form of Q-learning, a new unconnected experience wasdrawn from the replay memory to supply data for the next update. Because Q-learning
is an o↵-policy algorithm, it does not need to be applied along connected trajectories.
Q-learning with experience replay provided several advantages over the usual form of
Q-learning. The ability to use each stored experience for many updates allowed DQN tolearn more efficiently from its experiences. Experience replay reduced the variance of the
updates because successive updates were not correlated with one another as they would
be with standard Q-learning. And by removing the dependence of successive experienceson the current weights, experience replay eliminated one source of instability.
Mnih et al. modified standard Q-learning in a second way to improve its stability. As
in other methods that bootstrap, the target for a Q-learning update depends on thecurrent action-value function estimate. When a parameterized function approximation
method is used to represent action values, the target is a function of the same parametersthat are being updated. For example, the target in the update given by (16.3) ismaxa q̂(St+1 , a, wt ). Its dependence on wt complicates the process compared to thesimpler supervised-learning situation in which the targets do not depend on the parameters
being updated. As discussed in Chapter 11 this can lead to oscillations and/or divergence.
To address this problem Mnih et al. used a technique that brought Q-learning closer
to the simpler supervised-learning case while still allowing it to bootstrap. Whenevera certain number, C, of updates had been done to the weights w of the action-value
network, they inserted the network’s current weights into another network and held
these duplicate weights fixed for the next C updates of w. The outputs of this duplicate
network over the next C updates of w were used as the Q-learning targets. Letting q̃
denote the output of this duplicate network, then instead of (16.3) the update rule was:hi
wt+1 = wt + ↵ Rt+1 + max q̃(St+1 , a, wt ) q̂(St , At , wt ) rq̂(St , At , wt ).
a

A final modification of standard Q-learning was also found to improve stability. Theyclipped the error term Rt+1 + maxa q̃(St+1 , a, wt ) q̂(St , At , wt ) so that it remained in
the interval [ 1, 1].

16.6. Mastering the Game of Go

441

Mnih et al. conducted a large number of learning runs on 5 of the games to gain
insight into the e↵ect that various of DQN’s design features had on its performance.
They ran DQN with the four combinations of experience replay and the duplicate
target network being included or not included. Although the results varied from gameto game, each of these features alone significantly improved performance, and very
dramatically improved performance when used together. Mnih et al. also studied the role
played by the deep convolutional ANN in DQN’s learning ability by comparing the deep
convolutional version of DQN with a version having a network of just one linear layer, both
receiving the same stacked preprocessed video frames. Here, the improvement of the deep
convolutional version over the linear version was particularly striking across all 5 of the test
games.
Creating artificial agents that excel over a diverse collection of challenging tasks has
been an enduring goal of artificial intelligence. The promise of machine learning asa means for achieving this has been frustrated by the need to craft problem-specificrepresentations. DeepMind’s DQN stands as a major step forward by demonstratingthat a single agent can learn problem-specific features enabling it to acquire humancompetitive skills over a range of tasks. This demonstration did not produce one agent
that simultaneously excelled at all the tasks (because learning occurred separately foreach task), but it showed that deep learning can reduce, and possibly eliminate, the needfor problem-specific design and tuning. As Mnih et al. point out, however, DQN is not
a complete solution to the problem of task-independent learning. Although the skills
needed to excel on the Atari games were markedly diverse, all the games were played byobserving video images, which made a deep convolutional ANN a natural choice for thiscollection of tasks. In addition, DQN’s performance on some of the Atari 2600 games
fell considerably short of human skill levels on these games. The games most difficult
for DQN—especially Montezuma’s Revenge on which DQN learned to perform about aswell as the random player—require deep planning beyond what DQN was designed todo. Further, learning control skills through extensive practice, like DQN learned how toplay the Atari games, is just one of the types of learning humans routinely accomplish.
Despite these limitations, DQN advanced the state-of-the-art in machine learning byimpressively demonstrating the promise of combining reinforcement learning with modern
methods of deep learning.

16.6

Mastering the Game of Go

The ancient Chinese game of Go has challenged artificial intelligence researchers for many
decades. Methods that achieve human-level skill, or even superhuman-level skill, in othergames have not been successful in producing strong Go programs. Thanks to a very
active community of Go programmers and international competitions, the level of Go
program play has improved significantly over the years. Until recently, however, no Go
program had been able to play anywhere near the level of a human Go master.
A team at DeepMind (Silver et al., 2016) developed the program AlphaGo that broke
this barrier by combining deep ANNs (Section 9.7), supervised learning, Monte Carlo442

Chapter 16: Applications and Case Studies

tree search (MCTS, Section 8.11), and reinforcement learning. By the time of Silver etal.’s 2016 publication, AlphaGo had been shown to be decisively stronger than other Go
programs, and it had defeated the European Go champion Fan Hui 5 games to 0. These
were the first victories of a Go program over a professional human Go player without
handicap in full Go games. Shortly thereafter, a similar version of AlphaGo won stunning
victories over the 18-time world champion Lee Sedol, winning 4 out of a 5 games ina challenge match, making worldwide headline news. Artificial intelligence researchers
thought that it would be many more years, perhaps decades, before a program reachedthis level of play.
Here we describe AlphaGo and a successor program called AlphaGo Zero (Silver et al.
2017a). Where in addition to reinforcement learning, AlphaGo relied on supervised learning from a large database of expert human moves, AlphaGo Zero used only reinforcement
learning and no human data or guidance beyond the basic rules of the game (hence theZero in its name). We first describe AlphaGo in some detail in order to highlight therelative simplicity of AlphaGo Zero, which is both higher-performing and more of a purereinforcement learning program.
In many ways, both AlphaGo and AlphaGo Zero are descendants of Tesauro’s TDGammon (Section 16.1), itself a descendant of Samuel’s checkers player (Section 16.2).
All these programs included reinforcement learning over simulated games of self-play.
AlphaGo and AlphaGo Zero also built upon the progress made by DeepMind on playing
Atari games with the program DQN (Section 16.5) that used deep convolutional ANNs
to approximate optimal value functions.
Go is a game between two players who alternately place black and white ‘stones’ on unoccupied intersections, or ‘points,’ on a board witha grid of 19 horizontal and 19 vertical lines to
produce positions like that shown to the right.
The game’s goal is to capture an area of theboard larger than that captured by the opponent. Stones are captured according to simple
rules. A player’s stones are captured if they
are completely surrounded by the other player’s
stones, meaning that there is no horizontallyor vertically adjacent point that is unoccupied.
For example, the left panel of Figure 16.5 (onthe next page) shows three white stones withan unoccupied adjacent point (labeled X). If
black were to place a stone on X, then the three
A Go board configuration
white stones would be captured and taken o↵
the board (middle panel). However, if white were to place a stone on point X first, thenthe possibility of this capture would be blocked (right panel). Other rules are needed toprevent infinite capturing/recapturing loops. The game ends when neither player wishes
to place another stone. These rules are simple, but they produce a very complex gamethat has had wide appeal for thousands of years.

16.6. Mastering the Game of Go

443

X

Figure 16.5: Go capturing rule. Left: the three white stones are not surrounded because point
X is unoccupied. Middle: if black places a stone on X, the three white stones are captured andremoved from the board. Right: if white places a stone on point X first, the capture is blocked.

Methods that produce strong play for other games, such as chess, have not worked aswell for Go. The search space for Go is significantly larger than that of chess because
Go has a larger number of legal moves per position than chess (⇡ 250 versus ⇡ 35) and
Go games tend to involve more moves than chess games (⇡ 150 versus ⇡ 80). But thesize of the search space is not the major factor that makes Go so difficult. Exhaustive
search is infeasible for both chess and Go, and Go on smaller boards (e.g., 9 ⇥ 9) has
proven to be exceedingly difficult as well. Experts agree that the major stumbling block
to creating stronger-than-amateur Go programs is the difficulty of defining an adequate
position evaluation function. A good evaluation function allows search to be truncated ata feasible depth by providing relatively easy-to-compute predictions of what deeper searchwould likely yield. According to Müller (2002): “No simple yet reasonable evaluation
function will ever be found for Go.” A major step forward was the introduction of MCTS
to Go programs. The strongest programs at the time of AlphaGo’s development all
included MCTS, but master-level skill remained elusive.
Recall from Section 8.11 that MCTS is a decision-time planning procedure that does
not attempt to learn and store a global evaluation function. Like a rollout algorithm(Section 8.10), it runs many Monte Carlo simulations of entire episodes (here, entire
Go games) to select each action (here, each Go move: where to place a stone or toresign). Unlike a simple rollout algorithm, however, MCTS is an iterative procedure that
incrementally extends a search tree whose root node represents the current environment
state. As illustrated in Figure 8.10, each iteration traverses the tree by simulating
actions guided by statistics associated with the tree’s edges. In its basic version, whena simulation reaches a leaf node of the search tree, MCTS expands the tree by adding
some, or all, of the leaf node’s children to the tree. From the leaf node, or one of its
newly added child nodes, a rollout is executed: a simulation that typically proceeds all
the way to a terminal state, with actions selected by a rollout policy. When the rollout
completes, the statistics associated with the search tree’s edges that were traversed inthis iteration are updated by backing up the return produced by the rollout. MCTS
continues this process, starting each time at the search tree’s root at the current state, foras many iterations as possible given the time constraints. Then, finally, an action fromthe root node (which still represents the current environment state) is selected according
to statistics accumulated in the root node’s outgoing edges. This is the action the agent
takes. After the environment transitions to its next state, MCTS is executed again withthe root node set to represent the new current state. The search tree at the start of thisnext execution might be just this new root node, or it might include descendants of thisnode left over from MCTS’s previous execution. The remainder of the tree is discarded.

444

Chapter 16: Applications and Case Studies

16.6.1

AlphaGoThe main innovation that made AlphaGo such a strong player is that it selected moves by
a novel version of MCTS that was guided by both a policy and a value function learned
by reinforcement learning with function approximation provided by deep convolutional
ANNs. Another key feature is that instead of reinforcement learning starting from random
network weights, it started from weights that were the result of previous supervised
learning from a large collection of human expert moves.
The DeepMind team called AlphaGo’s modification of basic MCTS “asynchronous
policy and value MCTS,” or APV-MCTS. It selected actions via basic MCTS as describedabove but with some twists in how it extended its search tree and how it evaluated action
edges. In contrast to basic MCTS, which expands its current search tree by using stored
action values to select an unexplored edge from a leaf node, APV-MCTS, as implemented
in AlphaGo, expanded its tree by choosing an edge according to probabilities supplied bya 13-layer deep convolutional ANN, called the SL-policy network, trained previously bysupervised learning to predict moves contained in a database of nearly 30 million humanexpert moves.
Then, also in contrast to basic MCTS, which evaluates the newly-added state nodesolely by the return of a rollout initiated from it, APV-MCTS evaluated the node in two
ways: by this return of the rollout, but also by a value function, v✓ , learned previously bya reinforcement learning method. If s was the newly-added node, its value becamev(s) = (1⌘)v✓ (s) + ⌘G,(16.4)

where G was the return of the rollout and ⌘ controlled the mixing of the values resulting
from these two evaluation methods. In AlphaGo, these values were supplied by thevalue network, another 13-layer deep convolutional ANN that was trained as we describe
below to output estimated values of board positions. APV-MCTS’s rollouts in AlphaGo
were simulated games with both players using a fast rollout policy provided by a simple
linear network, also trained by supervised learning before play. Throughout its execution,
APV-MCTS kept track of how many simulations passed through each edge of the searchtree, and when its execution completed, the most-visited edge from the root node wasselected as the action to take, here the move AlphaGo actually made in a game.
The value network had the same structure as the deep convolutional SL policy network
except that it had a single output unit that gave estimated values of game positions
instead of the SL policy network’s probability distributions over legal actions. Ideally,
the value network would output optimal state values, and it might have been possible toapproximate the optimal value function along the lines of TD-Gammon described above:
self-play with nonlinear TD( ) coupled to a deep convolutional ANN. But the DeepMindteam took a di↵erent approach that held more promise for a game as complex as Go.
They divided the process of training the value network into two stages. In the first stage,
they created the best policy they could by using reinforcement learning to train an RL
policy network. This was a deep convolutional ANN with the same structure as the SL
policy network. It was initialized with the final weights of the SL policy network thatwere learned via supervised learning, and then policy-gradient reinforcement learning wasused to improve upon the SL policy. In the second stage of training the value network,16.6. Mastering the Game of Go445

the team used Monte Carlo policy evaluation on data obtained from a large number ofsimulated self-play games with moves selected by the RL policy network.
Figure 16.6 illustrates the networks used by AlphaGo and the steps taken to train themin what the DeepMind team called the “AlphaGo pipeline.” All these networks wereatrained before any live game play took place, and their weights remained fixed throughout
SL policy network
RL policy network
Value network
liveRollout
play. policy
pVnetwork
SL policy

U
RL policy p
Network

QT
Value Network

tio

n

]

cys
svioa
E
ln
ua

gorlieMR
CeP]

DataData

Su

ayy
Plla
elfP

Human expert positions

Slf
Se

ingn
arnio
Le at
ed ficrvis si
pe as
Su Cl

Cplaervis
ssied Lfic earati ninon g

PolicyPolicygradientgradient

Polic

Networks
Neural network

S
Rollout p
Policyb

Self-play positions

Figure 1 | Neural network training pipeline and architecture. a, A fast
the current player wins) in
Figure 16.6: AlphaGo pipeline. Adapted with permission from Macmillan Publishers Ltd:
supervised
learning(SL)
policynetworkp
areb,
Schematic representatio
rolloutNature,policyvol.
pπ andσ529(7587), p. 485, ©2016.
trained to predict human expert moves in a data set of positions.
AlphaGo. The policy netw
A reinforcement learning (RL) policy network pρ is initialized to the SLs as its input, passes it thro
Here is andsomeismore
detail about
AlphaGo’s
ANNs
and their
policy network,
then improvedby policy
gradient
learning
to training. σThe
(SLidenticallypolicy network) or ρ
structured
SL
and
RL
policynetworks
weresimilar
to
DQN’s
deep
convolutional
networkmaximize the outcome (that is, winning more games) against previous
distributionpσ (a | s) or pρ (adescribed
inSection16.5
forplayingAtarigames,
except
that
they
had
13
convolutional
probability map over the bversions of the policy network. A new data set is generated by playing
with thelayerconsistingof Finally,a soft-max
unitnetworkfor eachv point convolutionalon the 19 ⇥ 19
layers with pgameslayersof self-playwithfinal
the RL
policynetwork.
a valueθGoboard.
The
networks’
input
was
a
19
⇥
19
⇥
48
image
stack
in
which
each pointis trained by regression to predict the expected outcome (that is, whether
that predicts
the expectedon the Go board was represented by the values of 48 binary or integer-valued features.
For example, for each point, one feature indicated if the point was occupied by one ofsampledstate-action pairs (s, a), using stochastic gradient ascent to and its weights ρ are in
AlphaGo’s stones, one of its opponent’s stones, or was unoccupied, thus providing the
maximizelikelihood ofmove a selectedin features
state s were based
games“raw”therepresentationof thethe humanboard configuration.
Other
on between
the rules the curreprevious
iteration of theof Go, such as the number of adjacent points that were empty, the number of opponent
∂
log
p
(
a
|
s
)
σ
of
opponents
stones that would be ∆σcaptured∝ by placing a stone there, the number of turns since a stonein this way
was placed there, and other features
important.
current policy. We∂σ that the design team considered totobethe
Training the SL policy network took approximately 3 weeks using
a distributedtime steps
non-terminal
Weimplementation
trained a 13-layer
policy network,
whichonwe
the SL policyof stochastic
gradient ascent
50 call
processors.
The network
nalachieved
reward 57%
at the end of
network,from where
30 million
positionsfromthe KGS
Server.
The
accuracy,
the best
accuracy
achieved
by Go
other
groupsat netthe timeofpublication
player at time step t: +1
44.4%.expert
Training
the RL
network
waswithdoneanby
policy gradientreinforcement
work waspredicted
moves
on apolicy
held out
test set
accuracyof thenupdated at each tim
simulated
betweenthe only
RL policy
network’s
policy and57.0%learning
using allover
input
features,games
and 55.7%
using
raw board
posi- current
direction that maximizes
opponentsusing policies
randomly
selectedfrompolicies produced
tion andmove history
as inputs,
comparedto thestate-of-the-art
fromby earlier iterations
the learningalgorithm.
a randomly
selectedof opponents
24
otherofresearchgroupsof 44.4%Playing
at date againstof submission(full
resultscollectioninExtended Data Table 3). Small improvements in accuracy led to large
improvements in playing strength (Fig. 2a); larger networks achieve

∆

446

Chapter 16: Applications and Case Studies

prevented overfitting to the current policy. The reward signal was +1 if the current
policy won, 1 if it lost, and zero otherwise. These games directly pitted the two policies
against one another without involving MCTS. By simulating many games in parallel on
50 processors, the DeepMind team trained the RL policy network on a million games
in a single day. In testing the final RL policy, they found that it won more than 80%
of games played against the SL policy, and it won 85% of games played against a Go
program using MCTS that simulated 100,000 games per move.
The value network, whose structure was similar to that of the SL and RL policy
networks except for its single output unit, received the same input as the SL and RL
policy networks with the exception that there was an additional binary feature giving
the current color to play. Monte Carlo policy evaluation was used to train the networkfrom data obtained from a large number of self-play games played using the RL policy.
To avoid overfitting and instability due to the strong correlations between positions
encountered in self-play, the DeepMind team constructed a data set of 30 million positions
each chosen randomly from a unique self-play game. Then training was done using 50
million mini-batches each of 32 positions drawn from this data set. Training took one
week on 50 GPUs.
The rollout policy was learned prior to play by a simple linear network trained bysupervised learning from a corpus of 8 million human moves. The rollout policy network
had to output actions quickly while still being reasonably accurate. In principle, the SL
or RL policy networks could have been used in the rollouts, but the forward propagation
through these deep networks took too much time for either of them to be used in rollout
simulations, a great many of which had to be carried out for each move decision during
live play. For this reason, the rollout policy network was less complex than the otherpolicy networks, and its input features could be computed more quickly than the features
used for the policy networks. The rollout policy network allowed approximately 1,000
complete game simulations per second to be run on each of the processing threads that
AlphaGo used.
One may wonder why the SL policy was used instead of the better RL policy to selectactions in the expansion phase of APV-MCTS. These policies took the same amount oftime to compute because they used the same network architecture. The team actuallyfound that AlphaGo played better against human opponents when APV-MCTS used as
the SL policy instead of the RL policy. They conjectured that the reason for this wasthat the latter was tuned to respond to optimal moves rather than to the broader setof moves characteristic of human play. Interestingly, the situation was reversed for thevalue function used by APV-MCTS. They found that when APV-MCTS used the valuefunction derived from the RL policy, it performed better than if it used the value function
derived from the SL policy.
Several methods worked together to produce AlphaGo’s impressive playing skill. The
DeepMind team evaluated di↵erent versions of AlphaGo in order to assess the contributions
made by these various components. The parameter ⌘ in (16.4) controlled the mixingof game state evaluations produced by the value network and by rollouts. With ⌘ = 0,
AlphaGo used just the value network without rollouts, and with ⌘ = 1, evaluation
relied just on rollouts. They found that AlphaGo using just the value network played

16.6. Mastering the Game of Go

447

better than the rollout-only AlphaGo, and in fact played better than the strongest of allother Go programs existing at the time. The best play resulted from setting ⌘ = 0.5,
indicating that combining the value network with rollouts was particularly important
to AlphaGo’s success. These evaluation methods complemented one another: the value
network evaluated the high-performance RL policy that was too slow to be used in live
play, while rollouts using the weaker but much faster rollout policy were able to add
precision to the value network’s evaluations for specific states that occurred during games.
Overall, AlphaGo’s remarkable success fueled a new round of enthusiasm for the promiseof artificial intelligence, specifically for systems combining reinforcement learning withdeep ANNs, to address problems in other challenging domains.

16.6.2AlphaGo Zero

Building upon the experience with AlphaGo, a DeepMind team developed AlphaGo Zero
(Silver et al. 2017a). In contrast to AlphaGo, this program used no human data or
guidance beyond the basic rules of the game (hence the Zero in its name). It learned
exclusively from self-play reinforcement learning, with input giving just “raw” descriptionsof the placements of stones on the Go board. AlphaGo Zero implemented a form ofpolicy iteration (Section 4.3), interleaving policy evaluation with policy improvement.
Figure 16.7 is an overview of AlphaGo Zero’s algorithm. A significant di↵erence between
AlphaGo Zero and AlphaGo is that AlphaGo Zero used MCTS to select moves throughout
self-play reinforcement learning, whereas AlphaGo used MCTS for live play after—but notduring—learning. Other di↵erences besides not using any human data or human-craftedfeatures are that AlphaGo Zero used only one deep convolutional ANN and used a simpler
version of MCTS.
AlphaGo Zero’s MCTS was simpler than the version used by AlphaGo in that it did
not include rollouts of complete games, and therefore did not need a rollout policy. Each
iteration of AlphaGo Zero’s MCTS ran a simulation that ended at a leaf node of thecurrent search tree instead of at the terminal position of a complete game simulation.
But as in AlphaGo, each iteration of MCTS in AlphaGo Zero was guided by the output ofa deep convolutional network, labeled f✓ in Figure 16.7, where ✓ is the network’s weightvector. The input to the network, whose architecture we describe below, consisted of raw
representations of board positions, and its output had two parts: a scalar value, v, anestimate of the probability that the current player will win from from the current board
position, and a vector, p, of move probabilities, one for each possible stone placement on
the current board, plus the pass, or resign, move.
Instead of selecting self-play actions according to the probabilities p, however, AlphaGo
Zero used these probabilities, together with the network’s value output, to direct each
execution of MCTS, which returned new move probabilities, shown in Figure 16.7 as thepolicies ⇡i . These policies benefitted from the many simulations that MCTS conductedeach time it executed. The result was that the policy actually followed by AlphaGo
Zero was an improvement over the policy given by the network’s outputs p. Silver et al.
(2017a) wrote that “MCTS may therefore be viewed as a powerful policy improvement
operator.”

448

Chapter 16: Applications and Case Studies

Figure 16.7: AlphaGo Zero self-play reinforcement learning. a) The program played many
Figure1: Self-play
reinforcement
learning
in AlphaGo
Zero.
a The sprogram
plays a game
games
againstitself, one
shown here asa sequenceof board
positionsi , i = 1, 2, . . . , T , withmoves
= against
1, 2, . . .itself.
, T , andwinnerz. Eachai was determinedby actionis probabilities
i , isT
s1 ,a...,
In eachpositionst , amove
Monte-Carlotree search (MCTS)
executed (see ⇡ireturned by MCTS executed from root node si and guided by a deep convolutional network,2) fusing
the latest neural network f . Moves are selected according to the search probabilhere Figurelabeled✓ , with latest weights ✓. Shown here for just one position s but repeated for all
si , therawof boardsi (together
severalitiesnetwork’s
computedinputs
by the wereMCTS,
at representations⇠ t . The terminal
positionpositions
sT is scoredto computewith
the gamepast positions, though not shown here), and its outputs were vectors p of move probabilitiesb Neural
networksearches,training andin AlphaGo
Zero. The
neuralnetworkthe
takes
the raw boardthat winnerguided z.MCTS’s
forwardscalar valuesv thatestimatedprobabilityof thecurrent
playerwinningfromeachposition
s
.
b)
Deep
convolutional
networktraining.
Trainingiposition s as its input, passes it through many convolutional layers with parameters , and outputs
examples were randomly sampled steps from recent self-play games. Weights ✓ were updated
both thea vectorp, representing
a probability
distribution ⇡overmoves, anda scalar value
v, representto movepolicyvector p toward
the probabilities
returnedby MCTS,and toinclude thewinnersz
intheestimatedwin
probabilityv.
Reprinted
from
draft
of
Silveretal.
(2017a)withing the probability of the current player winning in position s. The neural network is trained on
permission of the authors and DeepMind.
randomly sampled steps from recent games of self-play, (s, , z). The parameters are updated so

as to maximise the similarity of the policy vector p to the search probabilities , and to minimise

Here is more detail about AlphaGo Zero’s ANN and how it was trained. The network
betweenwinnerv andconsisting
the game winner(see Equation
tooktheas error
input
a 19 ⇥the19predicted⇥ 17 image
stackof 17 zbinaryfeature1).planes. The first
8 feature planes were raw representations of the positions of the current player’s stones in
the current and seven past board configurations: a feature value was 1 if a player’s stone
was on the corresponding point, and was 0 otherwise. The next 8 feature planes similarly
coded the positions of the opponent’s stones.4 A final input feature plane had a constantvalue indicating the color of the current play: 1 for black; 0 for white. Because repetition
is not allowed in Go and one player is given some number of “compensation points” fornot getting the first move, the current board position is not a Markov state of Go. This
is why features describing past board positions and the color feature were needed.
The network was “two-headed,” meaning that after a number of initial layers, thenetwork split into two separate “heads” of additional layers that separately fed into two
sets of output units. In this case, one head fed 362 output units producing 192 + 1 move

16.6. Mastering the Game of Go

449

probabilities p, one for each possible stone placement plus pass; the other head fed justone output unit producing the scalar v, an estimate of the probability that the current
player will win from the current board position. The network before the split consisted of
41 convolutional layers, each followed by batch normalization, and with skip connections
added to implement residual learning by pairs of layers (see Section 9.7). Overall, moveprobabilities and values were computed by 43 and 44 layers respectively.
Starting with random weights, the network was trained by stochastic gradient descent(with momentum, regularization, and step-size parameter decreasing as training continues)
using batches of examples sampled uniformly at random from all the steps of the mostrecent 500,000 games of self-play with the current best policy. Extra noise was added
to the network’s output p to encourage exploration of all possible moves. At periodic
checkpoints during training, which Silver et al. (2017a) chose to be at every 1,000 training
steps, the policy output by the ANN with the latest weights was evaluated by simulating
400 games (using MCTS with 1,600 iterations to select each move) against the current
best policy. If the new policy won (by a margin set to reduce noise in the outcome), thenit became the best policy to be used in subsequent self-play. The network’s weights were
updated to make the network’s policy output p more closely match the policy returned
by MCTS, and to make its value output, v, more closely match the probability that thecurrent best policy wins from the board position represented by the network’s input.
The DeepMind team trained AlphaGo Zero over 4.9 million games of self-play, which
took about 3 days. Each move of each game was selected by running MCTS for 1,600
iterations, taking approximately 0.4 second per move. Network weights were updated over
700,000 batches each consisting of 2,048 board configurations. They then ran tournamentswith the trained AlphaGo Zero playing against the version of AlphaGo that defeated Fan
Hui by 5 games to 0, and against the version that defeated Lee Sedol by 4 games to 1.
They used the Elo rating system to evaluate the relative performances of the programs.
The di↵erence between two Elo ratings is meant to predict the outcome of games between
the players. The Elo ratings of AlphaGo Zero, the version of AlphaGo that played against
Fan Hui, and the version that played against Lee Sedol were respectively 4,308, 3,144,
and 3,739. The gaps in these Elo ratings translate into predictions that AlphaGo Zerowould defeat these other programs with probabilities very close to one. In a match of 100
games between AlphaGo Zero, trained as described, and the exact version of AlphaGothat defeated Lee Sedol held under the same conditions that were used in that match,
AlphaGo Zero defeated AlphaGo in all 100 games.
The DeepMind team also compared AlphaGo Zero with a program using an ANN withthe same architecture but trained by supervised learning to predict human moves in adata set containing nearly 30 million positions from 160,000 games. They found that thesupervised-learning player initially played better than AlphaGo Zero, and was better at
predicting human expert moves, but played less well after AlphaGo Zero was trained fora day. This suggested that AlphaGo Zero had discovered a strategy for playing that wasdi↵erent from how humans play. In fact, AlphaGo Zero discovered, and came to prefer,some novel variations of classical move sequences.
Final tests of AlphaGo Zero’s algorithm were conducted with a version having a larger
ANN and trained over 29 million self-play games, which took about 40 days, again starting

450

Chapter 16: Applications and Case Studies

with random weights. This version achieved an Elo rating of 5,185. The team pitted
this version of AlphaGo Zero against a program called AlphaGo Master , the strongest
program at the time, that was identical to AlphaGo Zero but, like AlphaGo, used humandata and features. AlphaGo Master ’s Elo rating was 4,858, and it had defeated thestrongest human professional players 60 to 0 in online games. In a 100 game match,
AlphaGo Zero with the larger network and more extensive learning defeated AlphaGo
Master 89 games to 11, thus providing a convincing demonstration of the problem-solving
power of AlphaGo Zero’s algorithm.
AlphaGo Zero soundly demonstrated that superhuman performance can be achievedby pure reinforcement learning, augmented by a simple version of MCTS, and deep ANNs
with very minimal knowledge of the domain and no reliance on human data or guidance.
We will surely see systems inspired by the DeepMind accomplishments of both AlphaGoand AlphaGo Zero applied to challenging problems in other domains.
Recently, yet a better program, AlphaZero, was described by Silver et al. (2017b) that
does not even incorporate knowledge of Go. AlphaZero is a general reinforcement learningalgorithm that improves over the world’s hitherto best programs in the diverse games ofGo, chess, and shogi.

16.7

Personalized Web Services

Personalizing web services such as the delivery of news articles or advertisements is one
approach to increasing users’ satisfaction with a website or to increase the yield of amarketing campaign. A policy can recommend content considered to be the best for eachparticular user based on a profile of that user’s interests and preferences inferred fromtheir history of online activity. This is a natural domain for machine learning, and inparticular, for reinforcement learning. A reinforcement learning system can improve a
recommendation policy by making adjustments in response to user feedback. One way
to obtain user feedback is by means of website satisfaction surveys, but for acquiring
feedback in real time it is common to monitor user clicks as indicators of interest in alink.
A method long used in marketing called A/B testing is a simple type of reinforcement
learning used to decide which of two versions, A or B, of a website users prefer. Because
it is non-associative, like a two-armed bandit problem, this approach does not personalize
content delivery. Adding context consisting of features describing individual users andthe content to be delivered allows personalizing service. This has been formalized as acontextual bandit problem (or an associative reinforcement learning problem, Section 2.9)
with the objective of maximizing the total number of user clicks. Li, Chu, Langford, andSchapire (2010) applied a contextual bandit algorithm to the problem of personalizing
the Yahoo! Front Page Today webpage (one of the most visited pages on the internet atthe time of their research) by selecting the news story to feature. Their objective was tomaximize the click-through rate (CTR), which is the ratio of the total number of clicks
all users make on a webpage to the total number of visits to the page. Their contextual
bandit algorithm improved over a standard non-associative bandit algorithm by 12.5%.
Theocharous, Thomas, and Ghavamzadeh (2015) argued that better results are possible

16.7. Personalized Web Services451

by formulating personalized recommendation as a Markov decision problem (MDP) withthe objective of maximizing the total number of clicks users make over repeated visits toa website. Policies derived from the contextual bandit formulation are greedy in the sensethat they do not take long-term e↵ects of actions into account. These policies e↵ectively
treat each visit to a website as if it were made by a new visitor uniformly sampled fromthe population of the website’s visitors. By not using the fact that many users repeatedly
visit the same websites, greedy policies do not take advantage of possibilities provided bylong-term interactions with individual users.
As an example of how a marketing strategy might take advantage of long-term userinteraction, Theocharous et al. contrasted a greedy policy with a longer-term policy fordisplaying ads for buying a product, say a car. The ad displayed by the greedy policy
might o↵er a discount if the user buys the car immediately. A user either takes the o↵er
or leaves the website, and if they ever return to the site, they would likely see the sameo↵er. A longer-term policy, on the other hand, can transition the user “down a sales
funnel” before presenting the final deal. It might start by describing the availability offavorable financing terms, then praise an excellent service department, and then, on thenext visit, o↵er the final discount. This type of policy can result in more clicks by a userover repeated visits to the site, and if the policy is suitably designed, more eventual sales.
Working at Adobe Systems Incorporated, Theocharous et al. conducted experiments
to see if policies designed to maximize clicks over the long term could in fact improveover short-term greedy policies. The Adobe Marketing Cloud, a set of tools that many
companies use to run digital marketing campaigns, provides infrastructure for automatinguser-targeted advertising and fund-raising campaigns. Actually deploying novel policies
using these tools entails significant risk because a new policy may end up performing
poorly. For this reason, the research team needed to assess what a policy’s performance
would be if it were to be actually deployed, but to do so on the basis of data collectedunder the execution of other policies. A critical aspect of this research, then, was o↵-policy
evaluation. Further, the team wanted to do this with high confidence to reduce therisk of deploying a new policy. Although high confidence o↵-policy evaluation was acentral component of this research (see also Thomas, 2015; Thomas, Theocharous, andGhavamzadeh, 2015), here we focus only on the algorithms and their results.
Theocharous et al. compared the results of two algorithms for learning ad recommendation policies. The first algorithm, which they called greedy optimization, had the goalof maximizing only the probability of immediate clicks. As in the standard contextual
bandit formulation, this algorithm did not take the long-term e↵ects of recommendations
into account. The other algorithm, a reinforcement learning algorithm based on an MDP
formulation, aimed at improving the number of clicks users made over multiple visits toa website. They called this latter algorithm life-time value (LTV) optimization. Bothalgorithms faced challenging problems because the reward signal in this domain is very
sparse because users usually do not click on ads, and user clicking is very random so that
returns have high variance.
Data sets from the banking industry were used for training and testing these algorithms.
The data sets consisted of many complete trajectories of customer interaction with abank’s website that showed each customer one out of a collection of possible o↵ers. If

452

Chapter 16: Applications and Case Studies

a customer clicked, the reward was 1, and otherwise it was 0. One data set contained
approximately 200,000 interactions from a month of a bank’s campaign that randomlyo↵ered one of 7 o↵ers. The other data set from another bank’s campaign contained
4,000,000 interactions involving 12 possible o↵ers. All interactions included customer
features such as the time since the customer’s last visit to the website, the number of theirvisits so far, the last time the customer clicked, geographic location, one of a collection ofinterests, and features giving demographic information.
Greedy optimization was based on a mapping estimating the probability of a click
as a function of user features. The mapping was learned via supervised learning fromone of the data sets by means of a random forest (RF) algorithm (Breiman, 2001). RF
algorithms have been widely used for large-scale applications in industry because they aree↵ective predictive tools that tend not to overfit and are relatively insensitive to outliers
and noise. Theocharous et al. then used the mapping to define an "-greedy policy that
selected with probability 1-" the o↵er predicted by the RF algorithm to have the highest
probability of producing a click, and otherwise selected from the other o↵ers uniformly at
random.
LTV optimization used a batch-mode reinforcement learning algorithm called fitted
Q iteration (FQI). It is a variant of fitted value iteration (Gordon, 1999) adapted to
Q-learning. Batch mode means that the entire data set for learning is available fromthe start, as opposed to the online mode of the algorithms we focus on in this book inwhich data are acquired sequentially while the learning algorithm executes. Batch-mode
reinforcement learning algorithms are sometimes necessary when online learning is notpractical, and they can use any batch-mode supervised learning regression algorithm,
including algorithms known to scale well to high-dimensional spaces. The convergence of
FQI depends on properties of the function approximation algorithm (Gordon, 1999). For
their application to LTV optimization, Theocharous et al. used the same RF algorithm
they used for the greedy optimization approach. Because in this case FQI convergence
is not monotonic, Theocharous et al. kept track of the best FQI policy by o↵-policy
evaluation using a validation training set. The final policy for testing the LTV approach
was the "-greedy policy based on the best policy produced by FQI with the initialaction-value function set to the mapping produced by the RF for the greedy optimization
approach.
To measure the performance of the policies produced by the greedy and LTV approaches,
Theocharous et al. used the CTR metric and a metric they called the LTV metric. These
metrics are similar, except that the LTV metric critically distinguishes between individual
website visitors:
Total # of Clicks
CTR =,Total # of Visits
LTV =Total # of Clicks.
Total # of Visitors

Figure 16.8 illustrates how these metrics di↵er. Each circle represents a user visit to
the site; black circles are visits at which the user clicks. Each row represents visits by
a particular user. By not distinguishing between visitors, the CTR for these sequences
is 0.35, whereas the LTV is 1.5. Because LTV is larger than CTR to the extent that16.8. Thermal Soaring453

individual users revisit the site, it is an indicator of how successful a policy is in encouraging
users to engage in extended interactions with the site.

Figure 16.8: Click through rate (CTR) versus life-time value (LTV). Each circle represents
a user visit; black circles are visits at which the user clicks. Adapted from Theocharous etal. (2015).

Testing the policies produced by the greedy and LTV approaches was done using
a high confidence o↵-policy evaluation method on a test data set consisting of realworld interactions with a bank website served by a random policy. As expected, results
showed that greedy optimization performed best as measured by the CTR metric, while
LTV optimization performed best as measured by the LTV metric. Furthermore—
although we have omitted its details—the high confidence o↵-policy evaluation method
provided probabilistic guarantees that the LTV optimization method would, with highprobability, produce policies that improve upon policies currently deployed. Assured bythese probabilistic guarantees, Adobe announced in 2016 that the new LTV algorithm
would be a standard component of the Adobe Marketing Cloud so that a retailer could
issue a sequence of o↵ers following a policy likely to yield higher return than a policythat is insensitive to long-term results.

16.8

Thermal Soaring

Birds and gliders take advantage of upward air currents—thermals—to gain altitude inorder to maintain flight while expending little, or no, energy. Thermal soaring, as thisbehavior is called, is a complex skill requiring responding to subtle environmental cues
to increase altitude by exploiting a rising column of air for as long as possible. Reddy,
Celani, Sejnowski, and Vergassola (2016) used reinforcement learning to investigate
thermal soaring policies that are e↵ective in the strong atmospheric turbulence usuallyaccompanying rising air currents. Their primary goal was to provide insight into thecues birds sense and how they use them to achieve their impressive thermal soaring
performance, but the results also contribute to technology relevant to autonomous gliders.
Reinforcement learning had previously been applied to the problem of navigating efficiently
to the vicinity of a thermal updraft (Woodbury, Dunn, and Valasek, 2014) but not to themore challenging problem of soaring within the turbulence of the updraft itself.
Reddy et al. modeled the soaring problem as a continuing MDP with discounting.
The agent interacted with a detailed model of a glider flying in turbulent air. They

454

Chapter 16: Applications and Case Studies

devoted significant e↵ort toward making the model generate realistic thermal soaring
conditions, including investigating several di↵erent approaches to atmospheric modeling.
A
B
For the learning experiments, air flow in a three-dimensional box with one kilometersides, one of which was at ground level, was modeled by a sophisticated physics-based
set of partial di↵erential equations involving air velocity, temperature, and pressure.
Introducing small random perturbations into the numerical simulation caused the model
to produce analogs of thermal updrafts and accompanying turbulence (Figure 16.9 Left)
Glider flight was modeled by aerodynamic equations involving velocity, lift, drag, andother factors governing powerless flight of a fixed-wing aircraft. Maneuvering the glider
involved changing its angle of attack (the angle between the glider’s wing and the directionof air flow) and its bank angle (Figure 16.9 Right).

A

C

B

Lift Lz

Dbank angleyLift L

Drag D

angle of attack

z glide angle↵

wing direction

velocity directionx

1. Snapshots
the verticalvelocity
the temperaturefieldsof(B)thein our numerical simulati
Figure 16.9: ThermalFig.soaring
model:ofLeft:snapshotof (A)
theandverticalvelocity field
field,
red andofblue
colorsindicateregions offlow.
largeRight:
upwarddiagramand downward flow, respectiv
Liftsimulated cube ofz air: invelocity
red (blue)isthe
aL region
large
upward(downward)
indicateand low
respectively.
that thehot and cold regions driveof powerless flight showing
bankregions
angle ofµ highand angle
oftemperature,attack ↵. Adapted
withNoticepermission
Fromangle with the basic physics of convection. (C) The force-body diagram of flight with no tcell, inbank
agreement
PNAS vol. 113(22), p. E4879,
2016, Reddy,
Celani, Sejnowski, and Vergassola, Learning to Soar
The figure also shows the bank angle μ (blue), the angle of attack α (green), and the glide angle γ (rein Turbulent Environments.
yCD

accessible by controlling the angle of attack. At small angles of attack, the glider moves fast but also sisinks more slowly. If the angle of attack is too high, at about 16°, the glider stalls, leading to a sudde
The interface between
agent
andfortheenvironment
requireddefining
fixedtheangle
of attackmostof the simulations(Results,
Controlthe
overagent’s
the Angle of Attack).

Lift L the agent receives from the environment, and the reward
actions, the state informationDsignal. Drag
By experimenting
with various possibilities, Reddy et al. decided that three actions
In the sequel, w
contributeand enoughmore exploratory
strategies
areeach for the angle of attack
and thesignificantlybank angle were
for their purposes:
increment
asoptimal. It sho
preferred.
or decrement the current bank angle and angle of attack by 5 and 2.5 , respectively, or2algorithm(as otheThe
SARSA
algorithm
finds
theoptimalpolicybyestimating
leave them unchanged. This
resultedangle
of attackin 3 possible actions. The bank angle was bounded
identifies
an
appr
for
every
state–action
pair
its
Q
functiondefinedastheexpected
glide
angle
to remain between
15 and +15
zwing.directionisskipped
only
fo
sum
of
futurerewards
given
thecurrent
state
s
and
the
actiona.
Because a goal of theirstudy was to try to determine what minimal set of sensory
velocity direction
At
each
step,
the
Q
functionisupdatedasfollows:
cues are necessary for e↵ective soaring, both to shed light on the cues birds might use for Resultsxsoaring and to minimize
the sensing complexity required for automated glider soaring, Sensorimotor CuesQðs,aÞ → Qðs,aÞ + ηðrβQðs′,a′Þ − Qðs, aÞÞ,the authors tried various sets ofsignals
as input
to +thereinforcement
learning [5]agent. aspects of the leaof the verticalvelocity
(A)
and
the
temperaturefields(B)inournumerical
simulationsof 3Dspace
Rayleigh–Bénard conv
They started by using state aggregation (Section 9.3) of a four-dimensional state
motor cues that th
where
r
is
the
received
rewardandη
isthelearning
rate.
The
ed and blue
colorsindicateregionsoflarge
upwardanddownward
flow,
respectively.
Forthetemperature field, twith dimensions giving local vertical wind speed, local vertical wind acceleration, torqueof the reward use
updatebetween
is Noticemadetheonline
and
does
not
any
prior
high and low
temperature,
respectively.
that
the
hot
and
coldrequire
regionsdrive
themodelupwardanddownwardbran
dependingon the di↵erence
verticalwindvelocities
at theleft
andright ofwing
state
and action
sp
the flowor theflight. Thisfeatureis particularly
relevant
inis, withoutwith the basic
physicsofconvection.
(C)
The
force-bodydiagramofflightwithnothrust,that
anyenginetips, and the local temperature. Each dimension was discretized into three bins: positive necessary to discre
in animals.
ows the bank angle μ (blue), themodelingangle of decision-making
attack α (green),processes
and the glide
angle γWhen(red).the(D)algoThe rangeof horizontal
lookuptable reps
rithm
is
close
to
convergence,theQfunction
approaches
theolling the angle of attack. At small angles of attack, the glider moves fast but also sinks fast, whereas
atlargeranglesaveraged overdifsolution to Bellman’s dynamic programming equations (12).

16.8. Thermal Soaring

455

high, negative high, and small. Results, described below, showed that only two of these
dimensions were critical for e↵ective soaring behavior.
The overall objective of thermal soaring is to gain as much altitude as possible fromeach rising column of air. Reddy et al. tried a straightforward reward signal that rewarded
the agent at the end of each episode based on the altitude gained over the episode, alarge negative reward signal if the glider touched the ground, and zero otherwise. They
found that learning was not successful with this reward signal for episodes of realistic
duration and that eligibility traces did not help. By experimenting with various reward
signals, they found that learning was best with a reward signal that at each time step
linearly combined the vertical wind velocity and vertical wind acceleration observed on
the previous time step.
Learning was by one-step Sarsa, with actions selected according to a soft-max distribution based on normalized action values. Specifically, the action probabilities werecomputed according to (13.2) with action preferences:h(s, a, ✓) =q̂(s, a, ✓) minb q̂(s, b, ✓),⌧ maxb q̂(s, b, ✓) minb q̂(s, b, ✓)where ✓ is a parameter vector with one component for each action and aggregated groupof states, and q̂(s, a, ✓) merely returned the component corresponding to s, a in the usual
way for state aggregation methods. The above equation forms the action preferencesby normalizing the approximate action values to the interval [0, 1] then dividing by ⌧ , apositive “temperature parameter.”3 As ⌧ increases, the probability of selecting an action
becomes less dependent on its preference; as ⌧ decreases toward zero, the probability of
selecting the most highly-preferred action approaches one, making the policy approach
the greedy policy. The temperature parameter ⌧ was initialized to 2.0 and incrementally
decreased to 0.2 during learning. Action preferences were computed from the current
estimates of the action values: the action with the maximum estimated action value wasgiven preference 1/⌧ , the action with the minimum estimated action value was given
preference 0, and the preferences of the other actions were scaled between these extremes.
The step-size and discount-rate parameters were fixed at 0.1 and 0.98 respectively.
Each learning episode took place with the agent controlling simulated flight in an
independently generated period of simulated turbulent air currents. Each episode lasted
2.5 minutes simulated with a 1 second time step. Learning e↵ectively converged after a
few hundred episodes. The left panel of Figure 16.10 shows a sample trajectory before
learning where the agent selects actions randomly. Starting at the top of the volume
shown, the glider’s trajectory is in the direction indicated by the arrow and quickly loses
altitude. Figure 16.10’s right panel is a trajectory after learning. The glider starts at thesame place (here appearing at the bottom of the volume) and gains altitude by spiraling
within the rising column of air. Although Reddy et al. found that performance varied
widely over di↵erent simulated periods of air flow, the number of times the glider touched
the ground consistently decreased to nearly zero as learning progressed.
After experimenting with di↵erent sets of features available to the learning agent, it
turned out that the combination of just vertical wind acceleration and torques worked
3 Reddy et al. described this slightly di↵erently, but our version is equivalent to theirs.

456

Chapter 16: Applications and Case Studies

Figure 16.10: Sample thermal soaring trajectories, with arrows showing the direction of(b)flight from the same(a)starting point (note that the altitude scales are shifted).
Left: before
learning: the agent selects actions randomly and the glider descends. Right: after learning:
the glider gains altitude by following a spiral trajectory. Adapted with permission from PNAS
vol. 113(22), p. E4879, 2016, Reddy, Celani, Sejnowski, and Vergassola, Learning to Soar in
Turbulent Environments.

best. The authors conjectured that because these features give information about thegradient of vertical wind velocity in two di↵erent directions, they allow the controller toselect between turning by changing the bank angle or continuing along the same course
by leaving the bank angle alone. This allows the glider to stay within a rising column ofair. Vertical wind velocity is indicative of the strength of the thermal but does not help
in staying within the flow. They found that sensitivity to temperature was of little help.
They also found that controlling the angle of attack is not helpful in staying within aparticular thermal, being useful instead for traveling between thermals when coveringlarge distances, as in cross-country gliding and bird migration.
Due to the fact that soaring in di↵erent levels of turbulence requires di↵erent policies,
training was done in conditions ranging from weak to strong turbulence. In strong
turbulence the rapidly changing wind and glider velocities allowed less time for thecontroller to react. This reduced the amount of control possible compared to what
was possible for maneuvering when fluctuations were weak. Reddy et al. examined thepolicies Sarsa learned under these di↵erent conditions. Common to policies learned in all
regimes were these features: when sensing negative wind acceleration, bank sharply in the
direction of the wing with the higher lift; when sensing large positive wind accelerationand no torque, do nothing. However, di↵erent levels of turbulence led to policy di↵erences.
Policies learned in strong turbulence were more conservative in that they preferred smallbank angles, whereas in weak turbulence, the best action was to turn as much as possible
by banking sharply. Systematic study of the bank angles preferred by the policies learned
under the di↵erent conditions led the authors to suggest that by detecting when vertical

16.8. Thermal Soaring

457

wind acceleration crosses a certain threshold the controller can adjust its policy to copewith di↵erent turbulence regimes.
Reddy et al. also conducted experiments to investigate the e↵ect of the discount-rate
parameter on the performance of the learned policies. They found that the altitude
gained in an episode increased as increased, reaching a maximum for = .99, suggestingthat e↵ective thermal soaring requires taking into account long-term e↵ects of control
decisions.
This computational study of thermal soaring illustrates how reinforcement learning
can further progress toward di↵erent kinds of objectives. Learning policies having
access to di↵erent sets of environmental cues and control actions contributes to boththe engineering objective of designing autonomous gliders and the scientific objective ofimproving understanding of the soaring skills of birds. In both cases, hypotheses resulting
from the learning experiments can be tested in the field by instrumenting real gliders4
and by comparing predictions with observed bird soaring behavior.

4 This work has recently been applied to real gliders. See Reddy, Wong-Ng, Celani, Sejnowski, andVergassola, “Glider soaring via reinforcement learning in the field.” Nature 562 :236–239, 2018.

Chapter 17

Frontiers
In this final chapter we touch on some topics that are beyond the scope of this book butthat we see as particularly important for the future of reinforcement learning. Many ofthese topics bring us beyond what is reliably known, and some bring us beyond the MDP
framework.

17.1

General Value Functions and Auxiliary Tasks

Over the course of this book, our notion of value function has become quite general.
With o↵-policy learning we allowed a value function to be conditional on an arbitrarytarget policy. Then in Section 12.8 we generalized discounting to a termination function: S 7! [0, 1], so that a di↵erent discount rate could be applied at each time step indetermining the return (12.17). This allowed us to express predictions about how muchreward we will get over an arbitrary, state-dependent horizon. The next, and perhaps
final, step is to generalize beyond rewards to permit predictions about arbitrary signals.
Rather than predicting the sum of future rewards, we might predict the sum of the future
values of a sound or color sensation, or of an internal, highly processed signal such as
another prediction. Whatever signal is added up in this way in a value-function-likeprediction, we call it the cumulant of that prediction. We formalize it in a cumulant
signal Ct 2 R. Using this, a general value function, or GVF, is written
"1
!
#
k
X Y
.
v⇡, ,C (s) = E(Si ) Ck+1 St = s, At:1 ⇠ ⇡ .
(17.1)
k=t

i=t+1

As with conventional value functions (such as v⇡ or q⇤ ) this is an ideal function that
we seek to approximate with a parameterized form, which we might continue to denotev̂(s,w), although of course there would have to be a di↵erent w for each prediction, thatis, for each choice of ⇡, , and C. Because a GVF has no necessary connection to reward,
it is perhaps a misnomer to call it a value function. You might simply call it a prediction
or, to make it more distinctive, a forecast (Ring, in preparation). Whatever it is called, it

460

Chapter 17: Frontiers

is in the form of a value function and thus can be learned in the usual ways using themethods developed in this book for learning approximate value functions. Along withthe learned predictions, we might also learn policies to maximize the predictions in theusual ways by Generalized Policy Iteration (Section 4.6) or by actor–critic methods. In
this way an agent could learn to predict and control great numbers of signals, not justlong-term reward.
Why might it be useful to predict and control signals other than long-term reward?
These are auxiliary tasks in that they are extra (in addition to) the main task ofmaximizing reward. One answer is that the ability to predict and control a diverse
multitude of signals can constitute a powerful kind of environmental model. As we saw
in Chapter 8, a good model can enable the agent to get reward more efficiently. It takes
a couple of further concepts to develop this answer clearly, so we postpone it to the next
section. First let’s consider two simpler ways in which a multitude of diverse predictions
can be helpful to a reinforcement learning agent.
One simple way in which auxiliary tasks can help on the main task is that they may
require some of the same representations as are needed on the main task. Some of theauxiliary tasks may be easier, with less delay and a clearer connection between actions
and outcomes. If good features can be found early on easy auxiliary tasks, then thosefeatures may significantly speed learning on the main task. There is no necessary reason
why this has to be true, but in many cases it seems plausible. For example, if you learn
to predict and control your sensors over short time scales, say seconds, then you mightplausibly come up with part of the idea of physical objects, which would then greatly
help with the prediction and control of long-term reward.
We might imagine an artificial neural network (ANN) in which the last layer is split
into multiple parts, or heads, each working on a di↵erent task. One head might produce
the approximate value function for the main task (with reward as its cumulant) whereas
the others would produce solutions to various auxiliary tasks. All heads could propagate
errors by stochastic gradient descent into the same body—the shared preceding partof the network—which would then try to form representations, in its next-to-last layer,
to support all the heads. Researchers have experimented with auxiliary tasks such aspredicting change in pixels, predicting the next time step’s reward, and predicting the
distribution of the return. In many cases this approach has been shown to greatly
accelerate learning on the main task (Jaderberg et al., 2017). Multiple predictions
have similarly been repeatedly proposed as a way of directing the construction of state
estimates (see Section 17.3).
Another simple way in which the learning of auxiliary tasks can improve performanceis best explained by analogy to the psychological phenomena of classical conditioning(Section 14.2). One way of understanding classical conditioning is that evolution has
built in a reflexive (non-learned) association to a particular action from the predictionof a particular signal. For example, humans and many other animals appear to have abuilt-in reflex to blink whenever their prediction of being poked in the eye exceeds somethreshold. The prediction is learned, but the association from prediction to eye closure
is built in, and thus the animal is saved many unprotected pokes in its eye. Similarly,
the association from fear to increased heart rate, or to freezing, may be built in. Agent17.2. Temporal Abstraction via Options

461

designers can do something similar, connecting by design (without learning) predictionsof specific events to predetermined actions. For example, a self-driving car that learns topredict whether going forward will produce a collision could be given a built-in reflex tostop, or to turn away, whenever the prediction is above some threshold. Or consider avacuum-cleaning robot that learned to predict whether it might run out of battery power
before returning to the charger and that reflexively headed back to the charger whenever
the prediction became non-zero. The correct prediction would depend on the size of thehouse, the room the robot was in, and the age of the battery, all of which would be hardfor the robot designer to know. It would be difficult for the designer to build in a reliablealgorithm for deciding whether to head back to the charger in sensory terms, but it might
be easy to do this in terms of the learned prediction. We foresee many possible ways
like this in which learned predictions might combine usefully with built-in algorithms forcontrolling behavior.
Finally, perhaps the most important role for auxiliary tasks is in moving beyond theassumption we have made throughout this book that the state representation is fixedand given to the agent. To explain this role, we first have to take a few steps back toappreciate the magnitude of this assumption and the implications of removing it. We dothat in Section 17.3.

17.2

Temporal Abstraction via Options

An appealing aspect of the MDP formalism is that it can be applied usefully to tasks
at many di↵erent time scales. It can be used to formalize the task of deciding whichmuscles to twitch to grasp an object, which airplane flight to take to arrive conveniently
at a distant city, and which job to take to lead a satisfying life. These tasks di↵er greatly
in their time scales, yet each can be usefully formulated as an MDP that can be solvedby planning or learning processes as described in this book. All involve interaction withthe world, sequential decision making, and a goal usefully conceived of as accumulating
rewards over time, and so all can be formulated as MDPs.
Although all these tasks can be formulated as MDPs, you might think that they cannot
be formulated as a single MDP. They involve such di↵erent time scales, such di↵erent
notions of choice and action! It would be no good, for example, to plan a flight across acontinent at the level of muscle twitches. Yet for other tasks—such as grasping objects,
throwing darts, or hitting a baseball—low-level muscle twitches may be just the rightlevel. People do all these things seamlessly without appearing to switch between levels.
Can the MDP framework be stretched to cover all the levels simultaneously?
Perhaps it can. One popular idea is to formalize an MDP at a detailed level, with asmall time step, yet enable planning at higher levels using extended courses of action that
correspond to many base-level time steps. To do this we need a notion of course of actionthat extends over many time steps and includes a notion of termination. A general way toformulate these two ideas is as a policy, ⇡, and a state-dependent termination function, ,as in GVFs. We define a pair of these as a generalized notion of action termed an option.
To execute an option ! = h⇡! , ! i at time t is to obtain the action to take, At , from⇡! (·|St ), then terminate at time t + 1 with probability 1! (St+1 ). If the option does

462

Chapter 17: Frontiers

not terminate at t + 1, then At+1 is selected from ⇡! (·|St+1 ), and the option terminates at
t + 2 with probability 1! (St+2 ), and so on until eventual termination. It is convenient
to consider low-level actions to be special cases of options—each action a corresponds
to an option h⇡! , ! i whose policy picks the action (⇡! (s) = a for all s 2 S) and whose
termination function is zero ( ! (s) = 0 for all s 2 S+ ). Options e↵ectively extend theaction space. The agent can either select a low-level action/option, terminating after one
time step, or select an extended option that might execute for many time steps before
terminating.
Options are designed so that they are interchangable with low-level actions. For
example, the notion of an action-value function q⇡ naturally generalizes to an optionvalue function that takes a state and option as input and returns the expected return
starting from that state, executing that option to termination, and thereafter following
the policy, ⇡. We can also generalize the notion of policy to a hierarchical policy that
selects from options rather than actions, where options, when selected, execute until
termination. With these ideas, many of the algorithms in this book can be generalized tolearn approximate option-value functions and hierarchical policies. In the simplest case,the learning process ‘jumps’ from option initiation to option termination, with an update
only occurring when an option terminates. More subtly, updates can be made on eachtime step, using “intra-option” learning algorithms, which in general require o↵-policy
learning.
Perhaps the most important generalization made possible by option ideas is that of theenvironmental model as developed in Chapters 3, 4, and 8. The conventional model of anaction is the state-transition probabilities and the expected immediate reward for taking
the action in each state. How do conventional action models generalize to option models?
For options, the appropriate model is again of two parts, one corresponding to the statetransition resulting from executing the option and one corresponding to the expectedcumulative reward along the way. The reward part of an option model, analogous to the
expected reward for state–action pairs (3.5), is⇤. ⇥r(s, !) = E R1 + R2 + 2 R3 + · · · + ⌧ 1 R⌧ S0 = s, A0:⌧ 1 ⇠ ⇡! , ⌧ ⇠ ! , (17.2)
for all options ! and all states s 2 S, where ⌧ is the random time step at which the option
terminates according to ! . Note the role of the overall discounting parameter in thisequation—discounting is according to , but termination of the option is according to! . The state-transition part of an option model is a little more subtle. This part ofthe model characterizes the probability of each possible resulting state (as in (3.4)), but
now this state may result after various numbers of time steps, each of which must be
discounted di↵erently. The model for option ! specifies, for each state s that ! mightstart executing in, and for each state s0 that ! might terminate in,1. X
p(s0 |s, !) =k=1

k

Pr{Sk = s0 , ⌧ = k | S0 = s, A0:k 1 ⇠ ⇡! , ⌧ ⇠ ! }.

(17.3)

Note that, because of the factor of k , this p(s0 |s, !) is no longer a transition probability
and no longer sums to one over all values of s0 . (Nevertheless, we continue to use the ‘|’
notation in p.)

17.2. Temporal Abstraction via Options

463

The above definition of the transition part of an option model allows us to formulate
Bellman equations and dynamic programming algorithms that apply to all options,
including low-level actions as a special case. For example, the general Bellman equation
for the state values of a hierarchical policy ⇡ is
"
#
X
X
0
0
v⇡ (s) =⇡(!|s) r(s, !) +p(s |s, !)v⇡ (s ) ,(17.4)s0!2⌦(s)

where ⌦(s) denotes the set of options available in state s. If ⌦(s) includes only the
low-level actions, then this equation reduces to a version of the usual Bellman equation(3.14), except of course is included in the new p (17.3) and thus does not appear.
Similarly, the corresponding planning algorithms also have no . For example, the value
iteration algorithm with options, analogous to (4.10), is
"
#
X
.
0
0
vk+1 (s) = max r(s, !) +p(s |s, !)vk (s ) , for all s 2 S.
!2⌦(s)s0

If ⌦(s) includes all the low-level actions available in each state s, then this algorithm
converges to the conventional v⇤ , from which the optimal policy can be computed.
However, it is particularly useful to plan with options when only a subset of the possible
options are considered (in ⌦(s)) in each state. Value iteration will then converge to thebest hierarchical policy limited to the restricted set of options. Although this policy may
be sub-optimal, convergence can be much faster because fewer options are considered
and because each option can jump over many time steps.
To plan with options, the agent must either be given the option models, or learn them.
One natural way to learn an option model is to formulate it as a collection of GVFs (asdefined in the preceding section) and then learn the GVFs using the methods presented
in this book. It is not difficult to see how this could be done for the reward part of theoption model. You merely choose one GVF’s cumulant to be the reward (Ct = Rt ), its
policy to be the option’s policy (⇡ = ⇡! ), and its termination function to be the discountrate times the option’s termination function ( (s) = · ! (s)). The true GVF thenequals the reward part of the option model, v⇡, ,C (s) = r(s, !), and the learning methods
described in this book can be used to approximate it. The state-transition part of theoption model is a little more complicated. You need to allocate one GVF for each statethat the option might terminate in. We don’t want these GVFs to accumulate anything
except when the option terminates, and then only when termination is in the appropriate
state. This can be achieved by choosing the cumulant of the GVF that predicts transition
to state s0 to be Ct = (1
! (St )) St =s0 . The GVF’s policy and termination functions
are chosen the same as for the reward part of the option model. The true GVF thenequals the s0 portion of the option’s state-transition model, v⇡, ,C (s) = p(s0 |s, !), andagain this book’s methods could be employed to learn it. Although each of these steps
is seemingly natural, putting them all together (including function approximation andother essential components) is quite challenging and beyond the current state of the art.

464

Chapter 17: Frontiers

Exercise 17.1 This section has presented options for the discounted case, but discounting
is arguably inappropriate for control when using function approximation (Section 10.4).
What is the natural Bellman equation for a hierarchical policy, analogous to (17.4), but
for the average reward setting (Section 10.3)? What are the two parts of the option
model, analogous to (17.2) and (17.3), for the average reward setting?
⇤

17.3

Observations and State

Throughout this book we have written the learned approximate value functions (andthe policies in Chapter 13) as functions of the environment’s state. This is a significant
limitation of the methods presented in Part I, in which the learned value function was
implemented as a table such that any value function could be exactly approximated;that case is tantamount to assuming that the state of the environment is completely
observed by the agent. But in many cases of interest, and certainly in the lives of all
natural intelligences, the sensory input gives only partial information about the state of
the world. Some objects may be occluded by others, or behind the agent, or miles away.
In these cases, potentially important aspects of the environment’s state are not directly
observable, and it is a strong, unrealistic, and limiting assumption to assume that thelearned value function is implemented as a table over the environment’s state space.
The framework of parametric function approximation that we developed in Part II is far
less restrictive and, arguably, no limitation at all. In Part II we retained the assumptionthat the learned value functions (and policies) are functions of the environment’s state,
but allowed these functions to be arbitrarily restricted by the parameterization. It issomewhat surprising and not widely recognized that function approximation includes
important aspects of partial observability. For example, if there is a state variable that is
not observable, then the parameterization can be chosen such that the approximate value
does not depend on that state variable. The e↵ect is just as if the state variable were notobservable. Because of this, all the results obtained for the parameterized case apply topartial observability without change. In this sense, the case of parameterized function
approximation includes the case of partial observability.
Nevertheless, there are many issues that cannot be investigated without a more explicit
treatment of partial observability. Although we cannot give them a full treatment here,
we can outline the changes that would be needed to do so. There are four steps.
First, we would change the problem. The environment would emit not its states, but
only observations—signals that depend on its state but, like a robot’s sensors, provide
only partial information about it. For convenience, without loss of generality, we assume
that the reward is a direct, known function of the observation (perhaps the observation isa vector, and the reward is one of its components). The environmental interaction would
then have no explicit states or rewards, but could simply be an alternating sequence ofactions At 2 A and observations Ot 2 O:
A0 , O1 , A1 , O2 , A2 , O3 , A3 , O4 , . . . ,going on forever (cf. Equation 3.1) or forming episodes each ending with a special terminal
observation.

17.3. Observations and State

465

Second, we can recover the idea of state as used in this book from the sequence ofobservations and actions. Let us use the word history, and the notation Ht , for an initial.
portion of the trajectory up to an observation: Ht = A0 , O1 , . . . , At 1 , Ot . The history
represents the most that we can know about the past without looking outside of thedata stream (because the history is the whole past data stream). Of course, the history
grows with t and can become large and unwieldy. The idea of state is that of a compact
summary of the history that is useful for predicting future sequences. To be a summaryof the history, a state must be a function of history, St = f (Ht ). The summary would be
informationally perfect if it retained all information about the history (and thus could
be used to predict futures as accurately as could be done from the full history). In thiscase, the state St and the function f are said to have the Markov property, and St is astate as we have used the term in this book. Let us henceforth call it a Markov state
to distinguish it from states that are summaries of the history but are not sufficient topredict all futures. In practice, the states of real agents will not be Markov but may
approach it as an ideal.
To be more explicit about the Markov property it is useful to formalize possible futures.
Let a test be any specific sequence of alternating actions and observations that mightoccur in the future. For example, a three-step test might be denoted ⌧ = a1 o1 a2 o2 a3 o3 .
The probability of this test given a specific history h is defined as.
p(⌧ |h) = Pr{Ot+1 = o1 , Ot+2 = o2 , Ot+3 = o3 | Ht = h, At = a1 , At+1 = a2 , At+2 = a3 }.(17.5)
Formally, f is Markov if and only if, for any test ⌧ , and for any histories h and h0 that
map to the same state under f , the test’s probabilities given the two histories are equal:f (h) = f (h0 ) ) p(⌧ |h) = p(⌧ |h0 ),for all h, h0 , ⌧ 2 {A ⇥ O}⇤ .

(17.6)A Markov state summarizes all the information in the history necessary for determining
any test’s probability. In fact, it summarizes all that is necessary for making any prediction,
including any GVF. It also summarizes all that is necessary for optimal behavior: if f is.
Markov, then there is always a deterministic function ⇡ such that choosing At = ⇡(f (Ht ))is an optimal policy.
The third step in extending reinforcement learning to partial observability is to deal
with certain computational considerations. As mentioned earlier, we want the state to becompact—relatively small compared to the history. (The identity function, for example,
is not a good f even though it is Markov, because the corresponding state St = Ht wouldgrow unboundedly with time.) In addition, we don’t really want a function f that takes
whole histories. Instead, we want an f that can be compactly implemented with anincremental, recursive update that computes St+1 from St , incorporating only the next
increment of data, At and Ot+1 :.
St+1 = u(St , At , Ot+1 ), for all t

0,(17.7)

with the first state S0 given. The function u is called the state-update function. For
example, if f were the identity (St = Ht ), then u would merely extend St by appending At
and Ot+1 to it. Given f , it is always possible to construct a corresponding u, but it may

466

Chapter 17: Frontiers

Model

A, R

PlannerAuOS

Policy,Value fn.

R

World

A

Figure 17.1: A conceptual agent architecture including a model, a planner, and a state-update
function. The world in this case receives actions A and emits observations O. The observations
and a copy of the action are used by the state-update function u to produce the new state. The
new state is input to the policy and value function, producing the next action, and is also input
to the planner (and to u). The information flows most responsible for learning are shown bydashed lines that pass diagonally across the boxes that they change. The reward R directly
changes the policy and value function. The action, reward, and state change the model, whichworks closely with the planner to also change the policy and value function. Note that theoperation of the planner can be decoupled from the agent–environment interaction, whereas theother processes should operate in lock step with this interaction to keep up with the arrival of
new data. Also note that the model and planner do not deal with observations directly, but onlywith the states produced by u, which can act as targets for model learning.

not be computationally convenient and, as in the identity example, it may not produce
a compact state. The state-update function is a central part of any agent architecturethat handles partial observability. It must be efficiently computable, as no actions or
predictions can be made until the state is available. An overall diagram of such an agent
architecture is given in Figure 17.1.
A common strategy for finding a Markov state is to look for something compact thatis recursively updatable and enables accurate short-term predictions. In fact, it is only
necessary to make accurate one-step predictions. An important fact is that, if an f
is incrementally updatable, then it is Markov if and only if all one-step tests can be
accurately predicted, that is, if and only if
f (h) = f (h0 ) ) Pr{Ot+1 = o|Ht = h, At = a} = Pr{Ot+1 = o|Ht = h0 , At = a}, (17.8)
for all h, h0 2 {A ⇥ O}⇤ , o 2 O and a 2 A. Accurate one-step predictions are informationally sufficient, together with the state-update function, to accurately predict theprobability of any test of any length. This can be done by iteratively and alternately
making one-step predictions and applying the state-update function. From the whole
tree of possibilities the exact probability of any test or the expectation of any GVF
can be determined. These observations have led many researchers to focus on one-step
predictions rather than directly on multi-step predictions such as GVFs. However, note

17.3. Observations and State

467

that determining long-term predictions from single-step predictions is exponentially complex in the length of the predictions. Moreover, one-step predictions can be iterated
to give accurate long-term predictions only if they are exact. If there is any error orapproximation in the one-step predictions, then it can compound to make the long-term
predictions wildly inaccurate. In practice this is often what happens.
An example of obtaining Markov states through a state-update function is provided
by the popular Bayesian approach known as Partially Observable MDPs, or POMDPs.
In this approach the environment is assumed to have a well defined latent state Xt that
underlies and produces the environment’s observations, but is never available to the agent(and is not to be confused with the state St used by the agent to make predictions anddecisions). The natural Markov state, St , for a POMDP is the distribution over the latentstates given the history, called the belief state. For concreteness, assume the usual case inwhich there are a finite number of hidden states, Xt 2 {1, 2, . . . , d}. Then the belief state.
is the vector St = st 2 [0, 1]d with components.
st [i] = Pr{Xt = i | Ht }, for all possible latent states i 2 {1, 2, . . . , d}.

(17.9)The belief state remains the same size (same number of components) even as t grows. It
can also be incrementally updated by Bayes’ rule, assuming complete knowledge of theinternal workings of the environment. Specifically, the ith component of the belief-state
update function isPd

x=1 s[x]p(i, o|x, a),0
x=1
x0 =1 s[x]p(x , o|x, a).
u(s, a, o)[i] = Pd

Pd

for all a 2 A, o 2 O,

(17.10)and for all belief states s with components s[x], where the four-argument p function here
is not the usual one for MDPs (as in Chapter 3), but the analogous one for POMDPs,.
in terms of the latent state: p(x0 , o|x, a) = Pr{Xt = x0 , Ot = o |Xt 1 = x, At 1 = a}. Thisapproach is popular in theoretical work and has many significant applications, but its
assumptions and computational complexity scale poorly, and we do not recommend it asan approach to artificial intelligence.
Another example of Markov states is provided by Predictive State Representations,or PSRs. PSRs address the weakness of the POMDP approach that the semantics of
its agent state St are grounded in the environment state, Xt , which is never observed
and thus is difficult to learn about. In PSRs and related approaches, the semantics ofthe agent state is instead grounded in predictions about future observations and actions,which are readily observable. In PSRs, a Markov state is defined as a d-vector of theprobabilities of d specially chosen “core” tests as defined above (17.5). The vector is thenupdated by a state-update function u that is analogous to Bayes rule, but with a semantics
grounded in observable data, which arguably makes it easier to learn. This approach has
been extended in many ways, including end-tests, compositional tests, powerful “spectral”
methods, and closed-loop and temporally abstract tests learned by TD methods. Some of
the best theoretical developments are for systems known as Observable Operator Models(OOMs) and Sequential Systems (Thon, 2017).
The fourth and final step in our brief outline of how to handle partial observability inreinforcement learning is to re-introduce approximation. As discussed in the introduction

468

Chapter 17: Frontiers

to Part II, to approach artificial intelligence ambitiously we must embrace approximation.
This is just as true for states as it is for value functions. We must accept and work with
an approximate notion of state. The approximate state will play the same role in ouralgorithms as before, so we continue to use the notation St for the state used by theagent, even though it may not be Markov.
Perhaps the simplest example of an approximate state is just the latest observation,.
St = Ot . Of course this approach cannot handle any hidden state information. It would.
be better to use the last k observations and actions, St = Ot , At 1 , Ot 1 , . . . , At k , forsome k 1, which can be achieved by a state-update function that just shifts the newdata in and the oldest data out. This kth-order history approach is still very simple,
but can greatly increase the agent’s capabilities compared to trying to use the single
immediate observation directly as the state.
What happens when the Markov property (17.8) is only approximately satisfied?
Unfortunately, long-term prediction performance can degrade dramatically when onestep predictions become even slightly inaccurate. Longer-term tests, GVFs, and stateupdate functions may or may not approximate better. The short-term and long-term
approximation objectives are just di↵erent, and there are no useful theoretical guarantees
at present.
Nevertheless, there are still reasons to think that the general idea outlined in thissection applies to the approximate case. The general idea is that a state that is good forsome predictions is also good for others—in particular, that a Markov state, sufficient for
one-step predictions, is also sufficient for all others. If we step back from that specificresult for the Markov case, the general idea is similar to what we discussed in Section 17.1
with multi-headed learning and auxiliary tasks. We discussed how representations thatwere good for the auxiliary tasks were often also good for the main task. Taken together,
these suggest an approach to both partial observability and representation learning inwhich multiple predictions are pursued and used to direct the construction of statefeatures. The guarantee provided by the perfect-but-impractical Markov property isreplaced by the heuristic that what’s good for some predictions may be good for others.
This approach scales well with computational resources. With a powerful computer we
could experiment with large numbers of predictions, perhaps favoring those that are mostsimilar to the ones of ultimate interest, that are easiest to learn reliably, or that satisfy
other criteria. It is important here to move beyond selecting the predictions manually.
The agent should do it. This would require a general language for predictions, so thatthe agent can systematically explore a large space of possible predictions, sifting throughthem for the ones that are most useful.
In particular, both POMDP and PSR approaches can be applied with approximate
states. The semantics of the state is often useful in forming the state-update function, as
it is in these two approaches and in the kth-order history approach. However, there is nota strong need for the state to be accurate with respect to its semantics in order to retain
useful information. Some approaches to state augmentation, such as Echo state networks(Jaeger, 2002), keep almost arbitrary information about the history and can neverthelessperform well. There are many possibilities, and we expect more work and ideas in thisarea. Learning the state-update function for an approximate state is a major part of therepresentation learning problem as it arises in reinforcement learning.

17.4. Designing Reward Signals

17.4

469

Designing Reward Signals

A major advantage of reinforcement learning over supervised learning is that reinforcement
learning does not rely on detailed instructional information: generating a reward signal
does not depend on knowledge of what the agent’s correct actions should be. But thesuccess of a reinforcement learning application strongly depends on how well the reward
signal frames the goal of the application’s designer and how well the signal assesses
progress in reaching that goal. For these reasons, designing a reward signal is a criticalpart of any application of reinforcement learning.
By designing a reward signal we mean designing the part of an agent’s environmentthat is responsible for computing each scalar reward Rt and sending it to the agent at
each time t. In our discussion of terminology at the end of Chapter 14, we said that Rt
is more like a signal generated inside an animal’s brain than it is like an object or eventin the animal’s external environment. The parts of our brains that generate these signals
for us evolved over millions of years to be well suited to the challenges our ancestors
had to face in their struggles to propagate their genes to future generations. We should
therefore not think that designing a good reward signal is always an easy thing to do!
One challenge is to design a reward signal so that as an agent learns, its behavior
approaches, and ideally eventually achieves, what the application’s designer actuallydesires. This can be easy if the designer’s goal is simple and easy to identify, such as
finding the solution to a well-defined problem or earning a high score in a well-defined
game. In cases like these, it is usual to reward the agent according to its success in solving
the problem or its success in improving its score. But some problems involve goals thatare difficult to translate into reward signals. This is especially true when the problem
requires the agent to skillfully perform a complex task or set of tasks, such as would be
required of a useful household robotic assistant. Further, reinforcement learning agentscan discover unexpected ways to make their environments deliver reward, some of whichmight be undesirable, or even dangerous. This is a longstanding and critical challenge forany method, like reinforcement learning, that is based on optimization. We discuss thisissue more in Section 17.6, the final section of this book.
Even when there is a simple and easily identifiable goal, the problem of sparse reward
often arises. Delivering non-zero reward frequently enough to allow the agent to achieve
the goal once, let alone to learn to achieve it efficiently from multiple initial conditions,can be a daunting challenge. State–action pairs that clearly deserve to trigger reward may
be few and far between, and rewards that mark progress toward a goal can be infrequentbecause progress is difficult or even impossible to detect. The agent may wander aimlessly
for long periods of time (what Minsky, 1961, called the “plateau problem”).
In practice, designing a reward signal is often left to an informal trial-and-error search
for a signal that produces acceptable results. If the agent fails to learn, learns too slowly,
or learns the wrong thing, then the designer tweaks the reward signal and tries again.
To do this, the designer judges the agent’s performance by criteria that he or she isattempting to translate into a reward signal so that the agent’s goal matches his or her
own. And if learning is too slow, the designer may try to design a non-sparse reward signalthat e↵ectively guides learning throughout the agent’s interaction with its environment.

470

Chapter 17: Frontiers

It is tempting to address the sparse reward problem by rewarding the agent for achieving
subgoals that the designer thinks are important way stations to the overall goal. But
augmenting the reward signal with well-intentioned supplemental rewards may lead theagent to behave di↵erently from what is intended; the agent may end up not achieving
the overall goal. A better way to provide such guidance is to leave the reward signal
alone and instead augment the value-function approximation with an initial guess of what
it should ultimately be, or augment it with initial guesses as to what certain parts of it
should be. For example, suppose we wants to o↵er v0 : S ! R as an initial guess at thetrue optimal value function v⇤ , and that we are using linear function approximation withfeatures x : S ! Rd . Then we would define the initial value function approximation as.
v̂(s,w) = w> x(s) + v0 (s),(17.11)and update the weights w as usual. If the initial weight vector is 0, then the initialvalue function will be v0 , but the asymptotic solution quality will be determined by thefeature vectors as usual. This initialization can also be done for arbitrary nonlinear
approximators and arbitrary forms of v0 , though it is not guaranteed to always acceleratelearning.
A particularly e↵ective approach to the sparse reward problem is the shaping technique introduced by the psychologist B. F. Skinner and described in Section 14.3. Thee↵ectiveness of this technique relies on the fact that sparse reward problems are not justproblems with the reward signal; they are also problems with an agent’s policy in thatit prevents the agent from frequently encountering rewarding states. Shaping involveschanging the reward signal as learning proceeds, starting from a reward signal that is
not sparse given the agent’s initial behavior, and gradually modifying it toward a reward
signal suited to the problem of original interest. Shaping might also involve modifying
the dynamics of the task as learning proceeds. Each modification is made so that theagent is frequently rewarded given its current behavior. The agent faces a sequence ofincreasingly-difficult reinforcement learning problems, where what is learned at each stage
makes the next-harder problem relatively easy because the agent now encounters reward
more frequently than it would if it did not have prior experience with easier problems.
This kind of shaping is an essential technique in training animals, and it is e↵ective incomputational reinforcement learning as well.
What if you have no idea what the rewards should be, but there is another agent,perhaps a person, who is already expert at the task and whose behavior can be observed?
In this case you could use methods known variously as “imitation learning,” “learningfrom demonstration,” and “apprenticeship learning.” The idea here is to benefit fromthe expert agent but leave open the possibility of eventually performing better. Learningfrom an expert’s behavior can be done either by learning directly by supervised learning
or by extracting a reward signal using what is known as “inverse reinforcement learning”
and then using a reinforcement learning algorithm with that reward signal to learn a
policy. The task of inverse reinforcement learning as explored by Ng and Russell (2000)
is to try to recover the expert’s reward signal from the expert’s behavior alone. Thiscannot be done exactly because a policy can be optimal with respect to many di↵erent
reward signals (for example, all policies are optimal with respect to a constant reward
signal), but it is possible to find plausible reward signal candidates. Unfortunately, strong

17.4. Designing Reward Signals

471

assumptions are required, including knowledge of the environment’s dynamics and of thefeature vectors in which the reward signal is linear. The method also requires completely
solving the problem (e.g., by dynamic programming methods) multiple times. Thesedifficulties notwithstanding, Abbeel and Ng (2004) argue that the inverse reinforcement
learning approach can sometimes be more e↵ective than supervised learning for benefitingfrom the behavior of an expert.
Another approach to finding a good reward signal is to automate the trial-and-error
search for a good signal that we mentioned above. From an application perspective, thereward signal is a parameter of the learning algorithm. As is true for other algorithm
parameters, the search for a good reward signal can be automated by defining a space offeasible candidates and applying an optimization algorithm. The optimization algorithm
evaluates each candidate reward signal by running the reinforcement learning system withthat signal for some number of steps, and then scoring the overall result by a “high-level”
objective function intended to encode the designer’s true goal, ignoring the limitationsof the agent. Reward signals can even be improved via online gradient ascent, where
the gradient is that of the high-level objective function (Sorg, Lewis, and Singh, 2010).
Relating this approach to the natural world, the algorithm for optimizing the high-levelobjective function is analogous to evolution, where the high-level objective function is an
animal’s evolutionary fitness determined by the number of its o↵spring that survive toreproductive age.
Computational experiments with this bilevel optimization approach—one level analogous to evolution, and the other due to reinforcement learning by individual agents—haveconfirmed that intuition alone is not always adequate to devise a good reward signal(Singh, Lewis, and Barto, 2009). The performance of a reinforcement learning agent asevaluated by the high-level objective function can be very sensitive to details of the agent’s
reward signal in subtle ways determined by the agent’s limitations and the environment
in which it acts and learns. These experiments have also demonstrated that an agent’s
goal should not always be the same as the goal of the agent’s designer.
At first this seems counterintuitive, but it may be impossible for the agent to achieve thedesigner’s goal no matter what its reward signal is. The agent has to learn under various
kinds of constraints, such as limited computational power, limited access to information
about its environment, or limited time to learn. When there are constraints like these,learning to achieve a goal that is di↵erent from the designer’s goal can sometimes end up
getting closer to the designer’s goal than if that goal were pursued directly (Sorg, Singh,
and Lewis, 2010; Sorg, 2011). Examples of this in the natural world are easy to find.
Because we cannot directly assess the nutritional value of most foods, evolution—thedesigner of our reward signal—gave us a reward signal that makes us seek certain tastes.
Though certainly not infallible (indeed, possibly detrimental in environments that di↵er incertain ways from ancestral environments), this compensates for many of our limitations:
our limited sensory abilities, the limited time over which we can learn, and the risks
involved in finding a healthy diet through personal experimentation. Similarly, because
an animal cannot always observe its own evolutionary fitness, that objective function
does not work as a reward signal for learning. Evolution instead provides reward signalsthat are sensitive to observable predictors of evolutionary fitness.
Finally, remember that a reinforcement learning agent is not necessarily like a complete

472

Chapter 17: Frontiersorganism or robot; it can be a component of a larger behaving system. This meansthat reward signals may be influenced by things inside the larger behaving agent, suchas motivational states, memories, ideas, or even hallucinations. Reward signals may
also depend on properties of the learning process itself, such as measures of how muchprogress learning is making. Making reward signals sensitive to information about internal
factors such as these makes it possible for an agent to learn how to control the “cognitive
architecture” of which it is a part, as well as to acquire knowledge and skills that would be
difficult to learn from a reward signal that depended only on external events. Possibilities
like these led to the idea of “intrinsically-motivated reinforcement learning” that webriefly discuss further at the end of the following section.

17.5

Remaining Issues

In this book we have presented the foundations of a reinforcement learning approach toartificial intelligence. Roughly speaking, that approach is based on model-free and modelbased methods working together, as in the Dyna architecture of Chapter 8, combinedwith function approximation as developed in Part II. The focus has been on online andincremental algorithms, which we see as fundamental even to model-based methods, andon how these can be applied in o↵-policy training situations. The full rationale for thelatter has been presented only in this last chapter. That is, we have all along presented o↵policy learning as an appealing way to deal with the explore/exploit dilemma, but only inthis chapter have we discussed learning about many diverse auxiliary tasks simultaneouslywith GVFs and learning about the world hierarchically in terms of temporally-abstract
option models, both of which involve o↵-policy learning. Much remains to be worked
out, as we have indicated throughout the book and as evidenced by the directions for
additional research discussed in this chapter. But suppose we are generous and grant thebroad outlines of everything that we have done in the book and everything that has beenoutlined so far in this chapter. What would remain after that? Of course we can’t know
for sure what will be required, but we can make some guesses. In this section we highlight
six further issues which it seems to us will still need to be addressed by future research.
First, we still need powerful parametric function approximation methods that work wellin fully incremental and online settings. Methods based on deep learning and ANNs area major step in this direction but, still, only work well with batch training on large data
sets, with training from extensive o↵-line self play, or with learning from the interleaved
experience of multiple agents on the same task. These and other settings are ways ofworking around a basic limitation of today’s deep learning methods, which struggle tolearn rapidly in the incremental, online settings that are most natural for the reinforcement
learning algorithms emphasized in this book. The problem is sometimes described as
one of “catastrophic interference” or “correlated data.” When something new is learned
it tends to replace what has previously been learned rather than adding to it, with theresult that the benefit of the older learning is lost. Techniques such as “replay bu↵ers”
are often used to retain and replay old data so that its benefits are not permanently lost.
An honest assessment has to be that current deep learning methods are not well suited toonline learning. We see no reason that this limitation is insurmountable, but algorithms

17.5. Remaining Issues

473

that address it, while at the same time retaining the advantages of deep learning, have
not yet been devised. Most current deep learning research is directed toward working
around this limitation rather than removing it.
Second (and perhaps closely related), we still need methods for learning features suchthat subsequent learning generalizes well. This issue is an instance of a general problem
variously called “representation learning,” “constructive induction,” and “meta-learning”—
how can we use experience not just to learn a given desired function, but to learn inductive
biases such that future learning generalizes better and is thus faster? This is an old
problem, dating back to the origins of artificial intelligence and pattern recognition in
the 1950s and 1960s.1 Such age should give one pause. Perhaps there is no solution. But
it is equally likely that the time for finding a solution and demonstrating its e↵ectivenesshas not yet arrived. Today machine learning is conducted at a far larger scale than it has
been in the past, and the potential benefits of a good representation learning method havebecome much more apparent. We note that a new annual conference—the International
Conference on Learning Representations—has been exploring this and related topics
every year since 2013. It is also less common to explore representation learning withina reinforcement learning context. Reinforcement learning brings some new possibilities
to this old issue, such as the auxiliary tasks discussed in Section 17.1. In reinforcement
learning, the problem of representation learning can be identified with the problem oflearning the state-update function discussed in Section 17.3.
Third, we still need scalable methods for planning with learned environment models.
Planning methods have proven extremely e↵ective in applications such as AlphaGo
Zero and computer chess in which the model of the environment is known from therules of the game or can otherwise be supplied by human designers. But cases of fullmodel-based reinforcement learning, in which the environment model is learned fromdata and then used for planning, are rare. The Dyna system described in Chapter 8 is
one example, but as described there and in most subsequent work it uses a tabular model
without function approximation, which greatly limits its applicability. Only a few studies
have included learned linear models, and even fewer have also explored the inclusion of
temporally-abstract models using options as discussed in Section 17.2.
More work is needed before planning with learned models can be e↵ective. For example,
the learning of the model needs to be selective because the scope of a model stronglya↵ects planning efficiency. If a model focuses on the key consequences of the mostimportant options, then planning can be efficient and rapid, but if a model includes
details of unimportant consequences of options that are unlikely to be selected, thenplanning may be almost useless. Environment models should be constructed judiciouslywith regard to both their states and dynamics with the goal of optimizing the planning
process. The various parts of the model should be continually monitored as to the degree
to which they contribute to, or detract from, planning efficiency. The field has not
yet addressed this complex of issues or designed model-learning methods that take into
account their implications.
1 Some would claim that deep learning solves this problem, for example, that DQN as described in
Section 16.5 illustrates a solution, but we are unconvinced. There is as yet little evidence that deeplearning alone solves the representation learning problem in a general and efficient way.

474

Chapter 17: Frontiers

A fourth issue that needs to be addressed in future research is that of automating the
choice of tasks on which an agent works and uses to structure its developing competence.
It is usual in machine learning for human designers to set the tasks that the learning
agent is expected to master. Because these tasks are known in advance and remain fixed,
they can be built into the learning algorithm code. However, looking ahead, we will want
the agent to make its own choices about what tasks it should try to master. These might
be subtasks of a specific overall task that is already known, or they might be intended tocreate building blocks that permit more efficient learning of many di↵erent tasks that theagent is likely to face in the future but which are currently unknown.
These tasks may be like the auxiliary tasks or the GVFs discussed in Section 17.1, or
tasks solved by options as discussed in Section 17.2. In forming a GVF, for example, whatshould the cumulant, the policy, and the termination function be? The current state of
the art is to select these manually, but far greater power and generality would come frommaking these task choices automatically, particularly when they derive from what theagent has previously constructed as a result of representation learning or experience withprevious subproblems. If GVF design is automated, then the design choices themselves
will have to be explicitly represented. Rather than the task choices being in the mindof the designer and built into the code, they will have to be in the machine itself insuch a way that they can be set and changed, monitored, filtered, and searched amongautomatically. Tasks could then be built hierarchically upon others much like features are
in an ANN. The tasks are the questions, and the contents of the ANN are the answers to
those questions. We expect there will need to be a full hierarchy of questions to matchthe hierarchy of answers provided by modern deep learning methods.
The fifth issue that we would like to highlight for future research is that of theinteraction between behavior and learning via some computational analog of curiosity.
In this chapter we have been imagining a setting in which many tasks are being learned
simultaneously, using o↵-policy methods, from the same stream of experience. The actions
taken will of course influence this stream of experience, which in turn will determine howmuch learning occurs and which tasks are learned. When reward is not available, or notstrongly influenced by behavior, the agent is free to choose actions that maximize insome sense the learning on the tasks, that is, to use some measure of learning progressas an internal or “intrinsic” reward, implementing a computational form of curiosity. In
addition to measuring learning progress, intrinsic reward can, among other possibilities,
signal the receipt of unexpected, novel, or otherwise interesting input, or can assess theagent’s ability to cause changes in its environment. Intrinsic reward signals generated inthese ways can be used by an agent to pose tasks for itself by defining auxiliary tasks,
GVFs, or options, as discussed above, so that skills learned in this way can contributeto the agent’s ability to master future tasks. The result is a computational analog ofsomething like play. Many preliminary studies of such uses of intrinsic reward signals
have been conducted, and exciting topics for future research remain in this general area.
A final issue that demands attention in future research is that of developing methods tomake it acceptably safe to embed reinforcement learning agents into physical environments.
This is one of the most pressing areas for future research, and we discuss it further in thefollowing section.

17.6. Reinforcement Learning and the Future of Artificial Intelligence

17.6

475

Reinforcement Learning and the Future of
Artificial Intelligence

When we were writing the first edition of this book in the mid-1990s, artificial intelligence
was making significant progress and was having an impact on society, though it was
mostly still the promise of artificial intelligence that was inspiring developments. Machinelearning was part of that outlook, but it had not yet become indispensable to artificial
intelligence. By today that promise has transitioned to applications that are changing the
lives of millions of people, and machine learning has come into its own as a key technology.
As we write this second edition, some of the most remarkable developments in artificial
intelligence have involved reinforcement learning, most notably “deep reinforcement
learning”—reinforcement learning with function approximation by deep artificial neuralnetworks. We are at the beginning of a wave of real-world applications of artificial
intelligence, many of which will include reinforcement learning, deep and otherwise, thatwill impact our lives in ways that are hard to predict.
But an abundance of successful real-world applications does not mean that true
artificial intelligence has arrived. Despite great progress in many areas, the gulf between
artificial intelligence and the intelligence of humans, and other animals, remains great.
Superhuman performance can be achieved in some domains, even formidable domains
like Go, but it remains a significant challenge to develop systems that are like us inbeing complete, interactive agents having general adaptability and problem-solving skills,
emotional sophistication, creativity, and the ability to learn quickly from experience.
With its focus on learning by interacting with dynamic environments, reinforcement
learning, as it develops over the future, will be a critical component of agents with theseabilities.
Reinforcement learning’s connections to psychology and neuroscience (Chapters 14
and 15) underscore its relevance to another longstanding goal of artificial intelligence:
shedding light on fundamental questions about the mind and how it emerges from thebrain. Reinforcement learning theory is already contributing to our understanding of
the brain’s reward, motivation, and decision-making processes, and there is good reason
to believe that through its links to computational psychiatry, reinforcement learningtheory will contribute to methods for treating mental disorders, including drug abuse
and addiction.
Another contribution that reinforcement learning can make over the future is as an
aid to human decision making. Policies derived by reinforcement learning in simulated
environments can advise human decision makers in such areas as education, healthcare,transportation, energy, and public-sector resource allocation. Particularly relevant is the
key feature of reinforcement learning that it takes long-term consequences of decisions
into account. This is very clear in games like backgammon and Go, where some of
the most impressive results of reinforcement learning have been demonstrated, but it
is also a property of many high-stakes decisions that a↵ect our lives and our planet.
Reinforcement learning follows related methods for advising human decision making thathave been developed in the past by decision analysts in many disciplines. With advanced
function approximation methods and massive computational power, reinforcement learning

476

Chapter 17: Frontiers

methods have the potential to overcome some of the difficulties of scaling up traditional
decision-support methods to larger and more complex problems.
The rapid pace of advances in artificial intelligence has led to warnings that artificial
intelligence poses serious threats to our societies, even to humanity itself. The renowned
scientist and artificial intelligence pioneer Herbert Simon anticipated the warnings we arehearing today in a presentation at the Earthware Symposium at CMU in 2000 (Simon,
2000). He spoke of the eternal conflict between the promise and perils of any new
knowledge, reminding us of the Greek myths of Prometheus, the idealized hero of modern
science, who stole fire from the gods for the benefit of mankind, and of Pandora, whosemythical box could be opened by a small and innocent action to release untold perils on
the world. While accepting that this conflict is inevitable, Simon urged us to recognizethat as designers of our future and not mere spectators, the decisions we make can tilt
the scale in Prometheus’ favor. This is certainly true for reinforcement learning, which
can benefit society but can also produce undesirable outcomes if it is carelessly deployed.
Thus, the safety of artificial intelligence applications involving reinforcement learning is a
topic that deserves careful attention.
A reinforcement learning agent can learn by interacting with either the real world orwith a simulation of some piece of the real world, or by a mixture of these two sources ofexperience. Simulators provide safe environments in which an agent can explore and learn
without risking real damage to itself or to its environment. In most current applications,policies are learned from simulated experience instead of direct interaction with thereal world. In addition to avoiding undesirable real-world consequences, learning fromsimulated experience can make virtually unlimited data available for learning, generally
at less cost than needed to obtain real experience, and because simulations typically run
much faster than real time, learning can often occur more quickly than if it relied on realexperience.
Nevertheless, the full potential of reinforcement learning requires reinforcement learning
agents to be embedded into the flow of real-world experience, where they act, explore,
and learn in our world, and not just in their worlds. After all, reinforcement learningalgorithms—at least those upon which we focus in this book—are designed to learn online,
and they emulate many aspects of how animals are able to survive in nonstationary andhostile environments. Embedding reinforcement learning agents in the real world can betransformative in realizing the promises of artificial intelligence to amplify and extend
human abilities.
A major reason for wanting a reinforcement learning agent to act and learn in the realworld is that it is often difficult, sometimes impossible, to simulate real-world experience
with enough fidelity to make the resulting policies, whether derived by reinforcement
learning or by other methods, work well—and safely—when directing real actions. Thisis especially true for environments whose dynamics depend on the behavior of humans,such as in education, healthcare, transportation, and public policy—domains that cansurely benefit from improved decision making. However, it is for real-world embedded
agents that warnings about potential dangers of artificial intelligence need to be heeded.
Some of these warnings are particularly relevant to reinforcement learning. Because
reinforcement learning is based on optimization, it inherits the plusses and minuses of all
optimization methods. On the minus side is the problem of devising objective functions,17.6. Reinforcement Learning and the Future of Artificial Intelligence477

or reward signals in the case of reinforcement learning, so that optimization produces
the desired results while avoiding undesirable results. We said in Section 17.4 thatreinforcement learning agents can discover unexpected ways to make their environments
deliver reward, some of which might be undesirable, or even dangerous. When we specify
what we want a system to learn only indirectly, as we do in designing a reinforcement
learning system’s reward signal, we will not know how closely the agent will fulfill our desire
until its learning is complete. This is hardly a new problem with reinforcement learning;
recognition of it has a long history in both literature and engineering. For example, in
Goethe’s poem “The Sorcerer’s Apprentice” (Goethe, 1878), the apprentice uses magic toenchant a broom to do his job of fetching water, but the result is an unintended flood dueto the apprentice’s inadequate knowledge of magic. In the engineering context, Norbert
Wiener, the founder of cybernetics, warned of this problem more than half a century ago
by relating the supernatural story of “The Monkey’s Paw” (Wiener, 1964): “... it grants
what you ask for, not what you should have asked for or what you intend” (p. 59). The
problem has also been discussed at length in a modern context by Nick Bostrom (2014).
Anyone having experience with reinforcement learning has likely seen their systems
discover unexpected ways to obtain a lot of reward. Sometimes the unexpected behavior
is good: it solves a problem in a nice new way. In other instances, what the agent
learns violates considerations that the system designer may never have thought about.
Careful design of reward signals is essential if an agent is to act in the real world with noopportunity for human vetting of its actions or means to easily interrupt its behavior.
Despite the possibility of unintended negative consequences, optimization has been used
for hundreds of years by engineers, architects, and others whose designs have positively
impacted the world. We owe much that is good in our environment to the applicationof optimization methods. Many approaches have been developed to mitigate the riskof optimization, such as adding hard and soft constraints, restricting optimization torobust and risk-sensitive policies, and optimizing with multiple objective functions. Someof these approaches have been adapted to reinforcement learning, and more research isneeded to address these concerns. The problem of ensuring that a reinforcement learning
agent’s goal is attuned to our own remains a challenge.
Another challenge if reinforcement learning agents are to act and learn in the real worldis not just about what they might learn eventually, but about how they will behave while
they are learning. How do you make sure that an agent gets enough experience to learn ahigh-performing policy, all the while not harming its environment, other agents, or itself(or more realistically, while keeping the probability of harm acceptably low)? This problem
is also not novel or unique to reinforcement learning. Risk management and mitigationfor embedded reinforcement learning is similar to what control engineers have had toconfront from the beginning of using automatic control in situations where a controller’s
behavior can have unacceptable, possibly catastrophic, consequences, as in the control ofan aircraft or a delicate chemical process. Control applications rely on careful systemmodeling, model validation, and extensive testing, and there is a highly-developed bodyof theory aimed at ensuring convergence and stability of adaptive controllers designed foruse when the dynamics of the system to be controlled are not fully known. Theoreticalguarantees are never iron-clad because they depend on the validity of the assumptions
underlying the mathematics, but without this theory, combined with risk-management

478

Chapter 17: Frontiers

and mitigation practices, automatic control—adaptive and otherwise—would not be as
beneficial as it is today in improving the quality, efficiency, and cost-e↵ectiveness ofprocesses on which we have come to rely. One of the most pressing areas for future
reinforcement learning research is to adapt and extend methods developed in control
engineering with the goal of making it acceptably safe to fully embed reinforcement
learning agents into physical environments.
In closing, we return to Simon’s call for us to recognize that we are designers of our
future and not simply spectators. By decisions we make as individuals, and by the
influence we can exert on how our societies are governed, we can work toward ensuringthat the benefits made possible by a new technology outweigh the harm it can cause.
There is ample opportunity to do this in the case of reinforcement learning, which canhelp improve the quality, fairness, and sustainability of life on our planet, but which
can also release new perils. A threat already here is the displacement of jobs caused
by applications of artificial intelligence. Still there are good reasons to believe that thebenefits of artificial intelligence can outweigh the disruption it causes. As to safety,
hazards possible with reinforcement learning are not completely di↵erent from thosethat have been managed successfully for related applications of optimization and control
methods. As reinforcement learning moves out into the real world in future applications,developers have an obligation to follow best practices that have evolved for similar
technologies, while at the same time extending them to make sure that Prometheus keeps
the upper hand.

Bibliographical and Historical Remarks
17.1

General value functions were first explicitly identified by Sutton and colleagues(Sutton, 1995a; Sutton et al., 2011; Modayil, White, and Sutton, 2013). Ring (inpreparation) developed an extensive thought experiment with GVFs (“forecasts”)
that has been influential despite not yet having been published.
The first demonstrations of multi-headed learning in reinforcement learning
were by Jaderberg et al. (2017). Bellemare, Dabney, and Munos (2017) showedthat predicting more things about the distribution of reward could significantly
accelerate learning to optimize its expectation, an instance of auxiliary tasks.
Many others have since taken up this line of research.
The general theory of classical conditioning as learned predictions together withbuilt-in, reflexive reactions to the predictions has not to our knowledge beenclearly articulated in the psychological literature. Modayil and Sutton (2014)
describe it as an approach to the engineering of robots and other agents, callingit “Pavlovian control” to allude to its roots in classical conditioning.

17.2

The formalization of temporally abstract courses of action as options was introduced by Sutton, Precup, and Singh (1999), building on prior work by Parr (1998)
and Sutton (1995a), and on classical work on Semi-MDPs (e.g., see Puterman,
1994). Precup’s (2000) PhD thesis developed option ideas fully. An important
limitation of these early works is that they did not treat the o↵-policy case

17.6. Reinforcement Learning and the Future of Artificial Intelligence479

with function approximation. Intra-option learning in general requires o↵-policy
learning, which could not be done reliably with function approximation at thattime. Although now we have a variety of stable o↵-policy learning methods using
function approximation, their combination with option ideas had not been significantly explored at the time of publication of this book. Barto and Mahadevan(2003) and Hengst (2012) review the options formalism and other approaches totemporal abstraction.
Using GVFs to implement option models has not previously been described. Ourpresentation uses the trick introduced by Modayil, White, and Sutton (2014) for
predicting signals at the termination of policies.
Among the few works that have learned option models with function approximation are those by Sorg and Singh (2010), and by Bacon, Harb, and Precup(2017).
The extension of options and option models to the average-reward setting has
not yet been developed in the literature.
17.3

A good presentation of the POMDP approach is given by Monahan (1982). PSRs
and tests were introduced by Littman, Sutton, and Singh (2002). OOMs wereintroduced by Jaeger (1997, 1998, 2000). Sequential Systems, which unify PSRs,
OOMs, and many other works, were introduced in the PhD thesis of Michael Thon
(2017; Thon and Jaeger, 2015). Extensions to networks of temporal relationships
were developed by Tanner (2006; Sutton and Tanner, 2005) and then extended
to options (Sutton, Rafols, and Koop, 2006).
The theory of reinforcement learning with a non-Markov state representation wasdeveloped explicitly by Singh, Jaakkola, and Jordan (1994; Jaakkola, Singh, andJordan, 1995). Early reinforcement learning approaches to partial observability
were developed by Chrisman (1992), McCallum (1993, 1995), Parr and Russell(1995), Littman, Cassandra, and Kaelbling (1995), and by Lin and Mitchell(1992).

17.4

Early e↵orts to include advice and teaching in reinforcement learning include
those by Lin (1992), Maclin and Shavlik (1994), Clouse (1996), and Clouse andUtgo↵ (1992).
Skinner’s shaping should not be confused with the “potential-based shaping”
technique introduced by Ng, Harada, and Russell (1999). Their technique has
been shown by Wiewiora (2003) to be equivalent to the simpler idea of providing
an initial approximation to the value function, as in (17.11).

17.5

We recommend the book by Goodfellow, Bengio, and Courville (2016) for discussion of today’s deep learning techniques. The problem of catastrophic interferencein ANNs was developed by McCloskey and Cohen (1989), Ratcli↵ (1990), andFrench (1999). The idea of a replay bu↵er was introduced by Lin (1992) and usedprominently in deep learning in the Atari game playing system (Section 16.5,
Mnih et al., 2013, 2015).

480

Chapter 17: Frontiers

Minsky (1961) was one of the first to identify the problem of representation
learning.
Among the few works to consider planning with learned, approximate models
are those by Kuvayev and Sutton (1996), Sutton, Szepesvari, Geramifard, andBowling (2008), Nouri and Littman (2009), and Hester and Stone (2012).
The need to be selective in model construction to avoid slowing planning is wellknown in artificial intelligence. Some of the classic work is by Minton (1990) andTambe, Newell, and Rosenbloom (1990). Hauskrecht, Meuleau, Kaelbling, Dean,
and Boutilier (1998) showed this e↵ect in MDPs with deterministic options.
Schmidhuber (1991a, b) proposed how something like curiosity would result if
reward signals were a function of how quickly an agent’s environment modelis improving. The empowerment function proposed by Klyubin, Polani, andNehaniv (2005) is an information-theoretic measure of an agent’s ability to control
its environment that can function as an intrinsic reward signal. Baldassarre andMirolli (2013) is a collection of contributions by researchers studying intrinsic
reward and motivation from both biological and computational perspectives,
including a perspective on “intrinsically-motivated reinforcement learning,” touse the term introduced by Singh, Barto, and Chentenez (2004). See also Oudeyer
and Kaplan (2007), Oudeyer, Kaplan, and Hafner (2007), and Barto (2013).

References
Abbeel, P., Ng, A. Y. (2004). Apprenticeship learning via inverse reinforcement learning. In
Proceedings of the 21st International Conference on Machine Learning. ACM, New York.
Abramson, B. (1990). Expected-outcome: A general model of static evaluation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 12 (2):182–193.
Adams, C. D. (1982). Variations in the sensitivity of instrumental responding to reinforcer
devaluation. The Quarterly Journal of Experimental Psychology, 34(2):77–98.
Adams, C. D., Dickinson, A. (1981). Instrumental responding following reinforcer devaluation.
The Quarterly Journal of Experimental Psychology, 33(2):109–121.
Adams, R. A., Huys, Q. J. M., Roiser, J. P. (2015). Computational Psychiatry: towards amathematically informed understanding of mental illness. Journal of Neurology, Neurosurgery& Psychiatry. doi:10.1136/jnnp-2015-310737
Agrawal, R. (1995). Sample mean based index policies with O(logn) regret for the multi-armed
bandit problem. Advances in Applied Probability, 27 (4):1054–1078.
Agre, P. E. (1988). The Dynamic Structure of Everyday Life. PhD thesis, Massachusetts
Institute of Technology, Cambridge MA. AI-TR 1085, MIT Artificial Intelligence Laboratory.
Agre, P. E., Chapman, D. (1990). What are plans for? Robotics and Autonomous Systems,
6(1-2):17–34.
Aizerman, M. A., Braverman, E. Í., Rozonoer, L. I. (1964). Probability problem of patternrecognition learning and potential functions method. Avtomat. i Telemekh, 25 (9):1307–1323.
Albus, J. S. (1971). A theory of cerebellar function. Mathematical Biosciences, 10(1-2):25–61.
Albus, J. S. (1981). Brain, Behavior, and Robotics. Byte Books, Peterborough, NH.
Aleksandrov, V. M., Sysoev, V. I., Shemeneva, V. V. (1968). Stochastic optimization of systems.
Izv. Akad. Nauk SSSR, Tekh. Kibernetika:14–19.
Amari, S. I. (1998). Natural gradient works efficiently in learning. Neural Computation,
10 (2):251–276.
An, P. C. E. (1991). An Improved Multi-dimensional CMAC Neural network: Receptive Field
Function and Placement. PhD thesis, University of New Hampshire, Durham.
An, P. C. E., Miller, W. T., Parks, P. C. (1991). Design improvements in associative memories for
cerebellar model articulation controllers (CMAC). Artificial Neural Networks, pp. 1207–1210,
Elsevier North-Holland. http://www.incompleteideas.net/papers/AnMillerParks1991.pdf
Anderson, C. W. (1986). Learning and Problem Solving with Multilayer Connectionist Systems.
PhD thesis, University of Massachusetts, Amherst.
Anderson, C. W. (1987). Strategy learning with multilayer connectionist representations. In
Proceedings of the 4th International Workshop on Machine Learning, pp. 103–114. Morgan
Kaufmann.

482

References

Anderson, C. W. (1989). Learning to control an inverted pendulum using neural networks. IEEE
Control Systems Magazine, 9 (3):31–37.
Anderson, J. A., Silverstein, J. W., Ritz, S. A., Jones, R. S. (1977). Distinctive features,categorical perception, and probability learning: Some applications of a neural model.
Psychological Review, 84(5):413–451.
Andreae, J. H. (1963). STELLA, A scheme for a learning machine. In Proceedings of the 2nd
IFAC Congress, Basle, pp. 497–502. Butterworths, London.
Andreae, J. H. (1969). Learning machines—a unified view. In A. R. Meetham and R. A. Hudson(Eds.), Encyclopedia of Information, Linguistics, and Control, pp. 261–270. Pergamon,
Oxford.
Andreae, J. H. (1977). Thinking with the Teachable Machine. Academic Press, London.
Andreae, J. H. (2017a). A model of how the brain learns: A short introduction to multiple
context associative learning (MCAL) and the PP system. Unpublished report.
Andreae, J. H. (2017b). Working memory for the associative learning of language. Unpublishedreport.
Andreae, J. H., Cashin, P. M. (1969). A learning machine with monologue. International
Journal of Man–Machine Studies, 1(1):1–20.
Arthur, W. B. (1991). Designing economic agents that act like human agents: A behavioral
approach to bounded rationality. The American Economic Review, 81 (2):353–359.
Asadi, K., Allen, C., Roderick, M., Mohamed, A. R., Konidaris, G., Littman, M. (2017). Mean
actor critic. ArXiv:1709.00503.
Atkeson, C. G. (1992). Memory-based approaches to approximating continuous functions. In
Sante Fe Institute Studies in the Sciences of Complexity, Proceedings Vol. 12, pp. 521–521.
Addison-Wesley.
Atkeson, C. G., Moore, A. W., Schaal, S. (1997). Locally weighted learning. Artificial Intelligence
Review, 11 :11–73.
Auer, P., Cesa-Bianchi, N., Fischer, P. (2002). Finite-time analysis of the multiarmed bandit
problem. Machine learning, 47(2-3):235–256.
Bacon, P. L., Harb, J., Precup, D. (2017). The option-critic architecture. In Proceedings of the
Association for the Advancement of Artificial Intelligence, pp. 1726–1734.
Baird, L. C. (1995). Residual algorithms: Reinforcement learning with function approximation.
In Proceedings of the 12th International Conference on Machine Learning, pp. 30–37. Morgan
Kaufmann.
Baird, L. C. (1999). Reinforcement Learning through Gradient Descent. PhD thesis, Carnegie
Mellon University, Pittsburgh PA.
Baird, L. C., Klopf, A. H. (1993). Reinforcement learning with high-dimensional, continuous
actions. Wright Laboratory, Wright-Patterson Air Force Base, Tech. Rep. WL-TR-93-1147.
Baird, L., Moore, A. W. (1999). Gradient descent for general reinforcement learning. In Advancesin Neural Information Processing Systems 11, pp. 968–974. MIT Press, Cambridge MA.
Baldassarre, G., Mirolli, M. (Eds.) (2013). Intrinsically Motivated Learning in Natural andArtificial Systems. Springer-Verlag, Berlin Heidelberg.
Balke, A., Pearl, J. (1994). Counterfactual probabilities: Computational methods, bounds
and applications. In Proceedings of the Tenth International Conference on Uncertainty in
Artificial Intelligence, pp. 46–54. Morgan Kaufmann.
Baras, D., Meir, R. (2007). Reinforcement learning, spike-time-dependent plasticity, and the
BCM rule. Neural Computation, 19(8):2245–2279.

References

483

Barnard, E. (1993). Temporal-di↵erence methods and Markov models. IEEE Transactions on
Systems, Man, and Cybernetics, 23(2):357–365.
Barreto, A. S., Precup, D., Pineau, J. (2011). Reinforcement learning using kernel-based
stochastic factorization. In Advances in Neural Information Processing Systems 24, pp. 720–
728. Curran Associates, Inc.
Bartlett, P. L., Baxter, J. (1999). Hebbian synaptic modifications in spiking neurons thatlearn. Technical report, Research School of Information Sciences and Engineering, Australian
National University.
Bartlett, P. L., Baxter, J. (2000). A biologically plausible and locally optimal learning algorithm
for spiking neurons. Rapport technique, Australian National University.
Barto, A. G. (1985). Learning by statistical cooperation of self-interested neuron-like computing
elements. Human Neurobiology, 4(4):229–256.
Barto, A. G. (1986). Game-theoretic cooperativity in networks of self-interested units. In
J. S. Denker (Ed.), Neural Networks for Computing, pp. 41–46. American Institute of Physics,
New York.
Barto, A. G. (1989). From chemotaxis to cooperativity: Abstract exercises in neuronal learning
strategies. In R. Durbin, R. Maill and G. Mitchison (Eds.), The Computing Neuron, pp. 73–98.
Addison-Wesley, Reading, MA.
Barto, A. G. (1990). Connectionist learning for control: An overview. In T. Miller, R. S. Sutton,
and P. J. Werbos (Eds.), Neural Networks for Control, pp. 5–58. MIT Press, Cambridge,
MA.
Barto, A. G. (1991). Some learning tasks from a control perspective. In L. Nadel and D. L. Stein(Eds.), 1990 Lectures in Complex Systems, pp. 195–223. Addison-Wesley, Redwood City, CA.
Barto, A. G. (1992). Reinforcement learning and adaptive critic methods. In D. A. White and
D. A. Sofge (Eds.), Handbook of Intelligent Control: Neural, Fuzzy, and Adaptive Approaches,
pp. 469–491. Van Nostrand Reinhold, New York.
Barto, A. G. (1995a). Adaptive critics and the basal ganglia. In J. C. Houk, J. L. Davis, andD. G. Beiser (Eds.), Models of Information Processing in the Basal Ganglia, pp. 215–232.
MIT Press, Cambridge, MA.
Barto, A. G. (1995b). Reinforcement learning. In M. A. Arbib (Ed.), Handbook of Brain Theory
and Neural Networks, pp. 804–809. MIT Press, Cambridge, MA.
Barto, A. G. (2011). Adaptive real-time dynamic programming. In C. Sammut and G. I Webb
(Eds.), Encyclopedia of Machine Learning, pp. 19–22. Springer Science and Business Media.
Barto, A. G. (2013). Intrinsic motivation and reinforcement learning. In G. Baldassarre and M.
Mirolli (Eds.), Intrinsically Motivated Learning in Natural and Artificial Systems, pp. 17–47.
Springer-Verlag, Berlin Heidelberg.
Barto, A. G., Anandan, P. (1985). Pattern recognizing stochastic learning automata. IEEE
Transactions on Systems, Man, and Cybernetics, 15(3):360–375.
Barto, A. G., Anderson, C. W. (1985). Structural learning in connectionist systems. In
Proceedings of the Seventh Annual Conference of the Cognitive Science Society, pp. 43–54.
Barto, A. G., Anderson, C. W., Sutton, R. S. (1982). Synthesis of nonlinear control surfaces bya layered associative search network. Biological Cybernetics, 43(3):175–185.
Barto, A. G., Bradtke, S. J., Singh, S. P. (1991). Real-time learning and control using
asynchronous dynamic programming. Technical Report 91-57. Department of Computerand Information Science, University of Massachusetts, Amherst.
Barto, A. G., Bradtke, S. J., Singh, S. P. (1995). Learning to act using real-time dynamic
programming. Artificial Intelligence, 72(1-2):81–138.

484

References

Barto, A. G., Du↵, M. (1994). Monte Carlo matrix inversion and reinforcement learning. In
Advances in Neural Information Processing Systems 6, pp. 687–694. Morgan Kaufmann, San
Francisco.
Barto, A. G., Jordan, M. I. (1987). Gradient following without back-propagation in layered
networks. In M. Caudill and C. Butler (Eds.), Proceedings of the IEEE First Annual
Conference on Neural Networks, pp. II629–II636. SOS Printing, San Diego.
Barto, A. G., Mahadevan, S. (2003). Recent advances in hierarchical reinforcement learning.
Discrete Event Dynamic Systems, 13 (4):341–379.
Barto, A. G., Singh, S. P. (1990). On the computational economics of reinforcement learning. In
Connectionist Models: Proceedings of the 1990 Summer School. Morgan Kaufmann.
Barto, A. G., Sutton, R. S. (1981a). Goal seeking components for adaptive intelligence: An
initial assessment. Technical Report AFWAL-TR-81-1070. Air Force Wright Aeronautical
Laboratories/Avionics Laboratory, Wright-Patterson AFB, OH.
Barto, A. G., Sutton, R. S. (1981b). Landmark learning: An illustration of associative search.
Biological Cybernetics, 42(1):1–8.
Barto, A. G., Sutton, R. S. (1982). Simulation of anticipatory responses in classical conditioning
by a neuron-like adaptive element. Behavioural Brain Research, 4(3):221–235.
Barto, A. G., Sutton, R. S., Anderson, C. W. (1983). Neuronlike elements that can solve
difficult learning control problems. IEEE Transactions on Systems, Man, and Cybernetics,
13(5):835–846. Reprinted in J. A. Anderson and E. Rosenfeld (Eds.), Neurocomputing:Foundations of Research, pp. 535–549. MIT Press, Cambridge, MA, 1988.
Barto, A. G., Sutton, R. S., Brouwer, P. S. (1981). Associative search network: A reinforcement
learning associative memory. Biological Cybernetics, 40(3):201–211.
Barto, A. G., Sutton, R. S., Watkins, C. J. C. H. (1990). Learning and sequential decision
making. In M. Gabriel and J. Moore (Eds.), Learning and Computational Neuroscience:
Foundations of Adaptive Networks, pp. 539–602. MIT Press, Cambridge, MA.
Baxter, J., Bartlett, P. L. (2001). Infinite-horizon policy-gradient estimation. Journal of Artificial
Intelligence Research, 15 :319–350.
Baxter, J., Bartlett, P. L., Weaver, L. (2001). Experiments with infinite-horizon, policy-gradientestimation. Journal of Artificial Intelligence Research, 15 :351–381.
Bellemare, M. G., Dabney, W., Munos, R. (2017). A distributional perspective on reinforcement
learning. ArXiv:1707.06887.
Bellemare, M. G., Naddaf, Y., Veness, J., Bowling, M. (2013). The arcade learning environment:
An evaluation platform for general agents. Journal of Artificial Intelligence Research,
47:253–279.
Bellemare, M. G., Veness, J., Bowling, M. (2012). Investigating contingency awareness using
Atari 2600 games. In Proceedings of the Twenty-Sixth AAAI Conference on Artificial
Intelligence, pp. 864–871. AAAI Press, Menlo Park, CA.
Bellman, R. E. (1956). A problem in the sequential design of experiments. Sankhya, 16:221–229.
Bellman, R. E. (1957a). Dynamic Programming. Princeton University Press, Princeton.
Bellman, R. E. (1957b). A Markov decision process. Journal of Mathematics and Mechanics,
6(5):679–684.
Bellman, R. E., Dreyfus, S. E. (1959). Functional approximations and dynamic programming.
Mathematical Tables and Other Aids to Computation, 13:247–251.
Bellman, R. E., Kalaba, R., Kotkin, B. (1963). Polynomial approximation—A new computational
technique in dynamic programming: Allocation processes. Mathematical Computation,
17:155–161.

References

485

Bengio, Y. (2009). Learning deep architectures for AI. Foundations and Trends in Machine
Learning, 2(1):1–27.
Bengio, Y., Courville, A. C., Vincent, P. (2012). Unsupervised feature learning and deep learning:
A review and new perspectives. CoRR 1, ArXiv:1206.5538.
Bentley, J. L. (1975). Multidimensional binary search trees used for associative searching.
Communications of the ACM, 18 (9):509–517.
Berg, H. C. (1975). Chemotaxis in bacteria. Annual review of biophysics and bioengineering,
4(1):119–136.
Berns, G. S., McClure, S. M., Pagnoni, G., Montague, P. R. (2001). Predictability modulates
human brain response to reward. The journal of neuroscience, 21(8):2793–2798.
Berridge, K. C., Kringelbach, M. L. (2008). A↵ective neuroscience of pleasure: reward in humans
and animals. Psychopharmacology, 199(3):457–480.
Berridge, K. C., Robinson, T. E. (1998). What is the role of dopamine in reward: hedonic
impact, reward learning, or incentive salience? Brain Research Reviews, 28(3):309–369.
Berry, D. A., Fristedt, B. (1985). Bandit Problems. Chapman and Hall, London.
Bertsekas, D. P. (1982). Distributed dynamic programming. IEEE Transactions on AutomaticControl, 27(3):610–616.
Bertsekas, D. P. (1983). Distributed asynchronous computation of fixed points. Mathematical
Programming, 27(1):107–120.
Bertsekas, D. P. (1987). Dynamic Programming: Deterministic and Stochastic Models. PrenticeHall, Englewood Cli↵s, NJ.
Bertsekas, D. P. (2005). Dynamic Programming and Optimal Control, Volume 1, third edition.
Athena Scientific, Belmont, MA.
Bertsekas, D. P. (2012). Dynamic Programming and Optimal Control, Volume 2: Approximate
Dynamic Programming, fourth edition. Athena Scientific, Belmont, MA.
Bertsekas, D. P. (2013). Rollout algorithms for discrete optimization: A survey. In Handbook of
Combinatorial Optimization, pp. 2989–3013. Springer, New York.
Bertsekas, D. P., Tsitsiklis, J. N. (1989). Parallel and Distributed Computation: Numerical
Methods. Prentice-Hall, Englewood Cli↵s, NJ.
Bertsekas, D. P., Tsitsiklis, J. N. (1996). Neuro-Dynamic Programming. Athena Scientific,
Belmont, MA.
Bertsekas, D. P., Tsitsiklis, J. N., Wu, C. (1997). Rollout algorithms for combinatorial optimization. Journal of Heuristics, 3 (3):245–262.
Bertsekas, D. P., Yu, H. (2009). Projected equation methods for approximate solution of large
linear systems. Journal of Computational and Applied Mathematics, 227 (1):27–50.
Bhat, N., Farias, V., Moallemi, C. C. (2012). Non-parametric approximate dynamic programming
via the kernel method. In Advances in Neural Information Processing Systems 25, pp. 386–394.
Curran Associates, Inc.
Bhatnagar, S., Sutton, R., Ghavamzadeh, M., Lee, M. (2009). Natural actor–critic algorithms.
Automatica, 45 (11).
Biermann, A. W., Fairfield, J. R. C., Beres, T. R. (1982). Signature table systems and learning.
IEEE Transactions on Systems, Man, and Cybernetics, 12(5):635–648.
Bishop, C. M. (1995). Neural Networks for Pattern Recognition. Clarendon, Oxford.
Bishop, C. M. (2006). Pattern Recognition and Machine Learning. Springer Science + BusinessMedia New York LLC.
Blodgett, H. C. (1929). The e↵ect of the introduction of reward upon the maze performance ofrats. University of California Publications in Psychology, 4:113–134.

486

References

Boakes, R. A., Costa, D. S. J. (2014). Temporal contiguity in associative learning: Iinterference
and decay from an historical perspective. Journal of Experimental Psychology: Animal
Learning and Cognition, 40(4):381–400.
Booker, L. B. (1982). Intelligent Behavior as an Adaptation to the Task Environment. PhD thesis,
University of Michigan, Ann Arbor.
Bostrom, N. (2014). Superintelligence: Paths, Dangers, Strategies. Oxford University Press.
Bottou, L., Vapnik, V. (1992). Local learning algorithms. Neural Computation, 4 (6):888–900.
Boyan, J. A. (1999). Least-squares temporal di↵erence learning. In Proceedings of the 16th
International Conference on Machine Learning, pp. 49–56.
Boyan, J. A. (2002). Technical update: Least-squares temporal di↵erence learning. Machine
Learning, 49(2):233–246.
Boyan, J. A., Moore, A. W. (1995). Generalization in reinforcement learning: Safely approximating the value function. In Advances in Neural Information Processing Systems 7, pp. 369–376.
MIT Press, Cambridge, MA.
Bradtke, S. J. (1993). Reinforcement learning applied to linear quadratic regulation. In Advancesin Neural Information Processing Systems 5, pp. 295–302. Morgan Kaufmann.
Bradtke, S. J. (1994). Incremental Dynamic Programming for On-Line Adaptive Optimal Control.
PhD thesis, University of Massachusetts, Amherst. Appeared as CMPSCI Technical Report
94-62.
Bradtke, S. J., Barto, A. G. (1996). Linear least–squares algorithms for temporal di↵erence
learning. Machine Learning, 22:33–57.
Bradtke, S. J., Ydstie, B. E., Barto, A. G. (1994). Adaptive linear quadratic control using policy
iteration. In Proceedings of the American Control Conference, pp. 3475–3479. American
Automatic Control Council, Evanston, IL.
Brafman, R. I., Tennenholtz, M. (2003). R-max – a general polynomial time algorithm fornear-optimal reinforcement learning. Journal of Machine Learning Research, 3 :213–231.
Breiman, L. (2001). Random forests. Machine Learning, 45 (1):5–32.
Breiter, H. C., Aharon, I., Kahneman, D., Dale, A., Shizgal, P. (2001). Functional imagingof neural responses to expectancy and experience of monetary gains and losses. Neuron,30(2):619–639.
Breland, K., Breland, M. (1961). The misbehavior of organisms. American Psychologist,
16(11):681–684.
Bridle, J. S. (1990). Training stochastic model recognition algorithms as networks can lead tomaximum mutual information estimates of parameters. In Advances in Neural Information
Processing Systems 2, pp. 211–217. Morgan Kaufmann, San Mateo, CA.
Broomhead, D. S., Lowe, D. (1988). Multivariable functional interpolation and adaptive networks.
Complex Systems, 2:321–355.
Bromberg-Martin, E. S., Matsumoto, M., Hong, S., Hikosaka, O. (2010). A pallidus-habenuladopamine pathway signals inferred stimulus values. Journal of Neurophysiology, 104(2):1068–
1076.
Browne, C.B., Powley, E., Whitehouse, D., Lucas, S.M., Cowling, P.I., Rohlfshagen, P., Tavener,
S., Perez, D., Samothrakis, S., Colton, S. (2012). A survey of monte carlo tree search methods.
IEEE Transactions on Computational Intelligence and AI in Games, 4 (1):1–43.
Brown, J., Bullock, D., Grossberg, S. (1999). How the basal ganglia use parallel excitatory
and inhibitory learning pathways to selectively respond to unexpected rewarding cues. The
Journal of Neuroscience, 19(23):10502–10511.

References

487

Bryson, A. E., Jr. (1996). Optimal control—1950 to 1985. IEEE Control Systems, 13(3):26–33.
Buchanan, B. G., Mitchell, T., Smith, R. G., Johnson, C. R., Jr. (1978). Models of learning
systems. Encyclopedia of Computer Science and technology, 11.
Buhusi, C. V., Schmajuk, N. A. (1999). Timing in simple conditioning and occasion setting: A
neural network approach. Behavioural Processes, 45(1):33–57.
Buşoniu, L., Lazaric, A., Ghavamzadeh, M., Munos, R., Babus̆ka, R., De Schutter, B. (2012).
Least-squares methods for policy iteration. In M. Wiering and M. van Otterlo (Eds.),
Reinforcement Learning: State-of-the-Art, pp. 75–109. Springer-Verlag Berlin Heidelberg.
Bush, R. R., Mosteller, F. (1955). Stochastic Models for Learning. Wiley, New York.
Byrne, J. H., Gingrich, K. J., Baxter, D. A. (1990). Computational capabilities of single
neurons: Relationship to simple forms of associative and nonassociative learning in aplysia.
In R. D. Hawkins and G. H. Bower (Eds.), Computational Models of Learning, pp. 31–63.
Academic Press, New York.
Calabresi, P., Picconi, B., Tozzi, A., Filippo, M. D. (2007). Dopamine-mediated regulation ofcorticostriatal synaptic plasticity. Trends in Neuroscience, 30(5):211–219.
Camerer, C. (2011). Behavioral Game Theory: Experiments in Strategic Interaction. Princeton
University Press.
Campbell, D. T. (1960). Blind variation and selective survival as a general strategy in knowledgeprocesses. In M. C. Yovits and S. Cameron (Eds.), Self-Organizing Systems, pp. 205–231.
Pergamon, New York.
Cao, X. R. (2009). Stochastic learning and optimization—A sensitivity-based approach. Annual
Reviews in Control, 33 (1):11–24.
Cao, X. R., Chen, H. F. (1997). Perturbation realization, potentials, and sensitivity analysis of
Markov processes. IEEE Transactions on Automatic Control, 42 (10):1382–1393.
Carlström, J., Nordström, E. (1997). Control of self-similar ATM call traffic by reinforcement
learning. In Proceedings of the International Workshop on Applications of Neural Networks
to Telecommunications 3, pp. 54–62. Erlbaum, Hillsdale, NJ.
Chapman, D., Kaelbling, L. P. (1991). Input generalization in delayed reinforcement learning:
An algorithm and performance comparisons. In Proceedings of the Twelfth International
Conference on Artificial Intelligence, pp. 726–731. Morgan Kaufmann, San Mateo, CA.
Chaslot, G., Bakkes, S., Szita, I., Spronck, P. (2008). Monte-Carlo tree search: A new frameworkfor game AI. In Proceedings of the Fourth AAAI Conference on Artificial Intelligence andInteractive Digital Entertainment (AIDE-08), pp. 216–217. AAAI Press, Menlo Park, CA.
Chow, C.-S., Tsitsiklis, J. N. (1991). An optimal one-way multigrid algorithm for discrete-time
stochastic control. IEEE Transactions on Automatic Control, 36(8):898–914.
Chrisman, L. (1992). Reinforcement learning with perceptual aliasing: The perceptual distinctions approach. In Proceedings of the Tenth National Conference on Artificial Intelligence,
pp. 183–188. AAAI/MIT Press, Menlo Park, CA.
Christensen, J., Korf, R. E. (1986). A unified theory of heuristic evaluation functions andits application to learning. In Proceedings of the Fifth National Conference on Artificial
Intelligence, pp. 148–152. Morgan Kaufmann.
Cichosz, P. (1995). Truncating temporal di↵erences: On the efficient implementation of TD( )for reinforcement learning. Journal of Artificial Intelligence Research, 2:287–318.
Ciosek, K., Whiteson, S. (2017). Expected policy gradients. ArXiv:1706.05374v1. A revised version appeared in Proceedings of the Annual Conference of the Association for the Advancementof Artificial Intelligence, pp. 2868–2875.
Ciosek, K., Whiteson, S. (2018). Expected policy gradients for reinforcement learning. ArXiv:
1801.03326.

488

References

Claridge-Chang, A., Roorda, R. D., Vrontou, E., Sjulson, L., Li, H., Hirsh, J., Miesenböck, G.
(2009). Writing memories with light-addressable reinforcement circuitry. Cell, 139(2):405–
415.
Clark, R. E., Squire, L. R. (1998). Classical conditioning and brain systems: the role of awareness.
Science, 280(5360):77–81.
Clark, W. A., Farley, B. G. (1955). Generalization of pattern recognition in a self-organizing
system. In Proceedings of the 1955 Western Joint Computer Conference, pp. 86–91.
Clouse, J. (1996). On Integrating Apprentice Learning and Reinforcement Learning TITLE2.
PhD thesis, University of Massachusetts, Amherst. Appeared as CMPSCI Technical Report
96-026.
Clouse, J., Utgo↵, P. (1992). A teaching method for reinforcement learning systems. In
Proceedings of the 9th International Workshop on Machine Learning, pp. 92–101. Morgan
Kaufmann.
Cobo, L. C., Zang, P., Isbell, C. L., Thomaz, A. L. (2011). Automatic state abstraction fromdemonstration. In Proceedings of the Twenty-Second International Joint Conference on
Artificial Intelligence, pp. 1243-1248. AAAI Press.
Connell, J. (1989). A colony architecture for an artificial creature. Technical Report AI-TR-1151.
MIT Artificial Intelligence Laboratory, Cambridge, MA.
Connell, M. E., Utgo↵, P. E. (1987). Learning to control a dynamic physical system. Computational intelligence, 3 (1):330–337.
Contreras-Vidal, J. L., Schultz, W. (1999). A predictive reinforcement model of dopamine neurons
for learning approach behavior. Journal of Computational Neuroscience, 6(3):191–214.
Coulom, R. (2006). Efficient selectivity and backup operators in Monte-Carlo tree search. In
Proceedings of the 5th International Conference on Computers and Games (CG’06), pp. 72–83.
Springer-Verlag Berlin, Heidelberg.
Courville, A. C., Daw, N. D., Touretzky, D. S. (2006). Bayesian theories of conditioning in achanging world. Trends in Cognitive Science, 10(7):294–300.
Craik, K. J. W. (1943). The Nature of Explanation. Cambridge University Press, Cambridge.
Cross, J. G. (1973). A stochastic learning model of economic behavior. The Quarterly Journalof Economics, 87 (2):239–266.
Crow, T. J. (1968). Cortical synapses and reinforcement: a hypothesis. Nature, 219(5155):736–
737.
Curtiss, J. H. (1954). A theoretical comparison of the efficiencies of two classical methods and aMonte Carlo method for computing one component of the solution of a set of linear algebraic
equations. In H. A. Meyer (Ed.), Symposium on Monte Carlo Methods, pp. 191–233. Wiley,
New York.
Cybenko, G. (1989). Approximation by superpositions of a sigmoidal function. Mathematics ofcontrol, signals and systems, 2(4):303–314.
Cziko, G. (1995). Without Miracles: Universal Selection Theory and the Second Darvinian
Revolution. MIT Press, Cambridge, MA.
Dabney, W. (2014). Adaptive step-sizes for reinforcement learning. PhD thesis, University of
Massachusetts, Amherst.
Dabney, W., Barto, A. G. (2012). Adaptive step-size for online temporal di↵erence learning. In
Proceedings of the Annual Conference of the Association for the Advancement of Artificial
Intelligence.
Daniel, J. W. (1976). Splines and efficiency in dynamic programming. Journal of Mathematical
Analysis and Applications, 54:402–407.
Dann, C., Neumann, G., Peters, J. (2014). Policy evaluation with temporal di↵erences: A survey
and comparison. Journal of Machine Learning Research, 15 :809–883.

References

489

Daw, N. D., Courville, A. C., Touretzky, D. S. (2003). Timing and partial observability in the
dopamine system. In Advances in Neural Information Processing Systems 15, pp. 99–106.
MIT Press, Cambridge, MA.
Daw, N. D., Courville, A. C., Touretzky, D. S. (2006). Representation and timing in theories ofthe dopamine system. Neural Computation, 18(7):1637–1677.
Daw, N. D., Niv, Y., Dayan, P. (2005). Uncertainty based competition between prefrontal anddorsolateral striatal systems for behavioral control. Nature Neuroscience, 8(12):1704–1711.
Daw, N. D., Shohamy, D. (2008). The cognitive neuroscience of motivation and learning. Social
Cognition, 26(5):593–620.
Dayan, P. (1991). Reinforcement comparison. In D. S. Touretzky, J. L. Elman, T. J. Sejnowski,
and G. E. Hinton (Eds.), Connectionist Models: Proceedings of the 1990 Summer School,
pp. 45–51. Morgan Kaufmann.
Dayan, P. (1992). The convergence of TD( ) for general . Machine Learning, 8(3):341–362.
Dayan, P. (2002). Matters temporal. Trends in Cognitive Sciences, 6 (3):105–106.
Dayan, P., Abbott, L. F. (2001). Theoretical Neuroscience: Computational and Mathematical
Modeling of Neural Systems. MIT Press, Cambridge, MA.
Dayan, P., Berridge, K. C. (2014). Model-based and model-free Pavlovian reward learning:
Revaluation, revision, and revaluation. Cognitive, A↵ective, & Behavioral Neuroscience,14(2):473–492.
Dayan, P., Niv, Y. (2008). Reinforcement learning: the good, the bad and the ugly. Current
Opinion in Neurobiology, 18(2):185–196.
Dayan, P., Niv, Y., Seymour, B., Daw, N. D. (2006). The misbehavior of value and the disciplineof the will. Neural Networks, 19 (8):1153–1160.
Dayan, P., Sejnowski, T. (1994). TD( ) converges with probability 1. Machine Learning,
14(3):295–301.
De Asis, K., Hernandez-Garcia, J. F., Holland, G. Z., Sutton, R. S. (2017). Multi-step Reinforcement Learning: A Unifying Algorithm. ArXiv:1703.01327.
de Farias, D. P. (2002). The Linear Programming Approach to Approximate Dynamic Programming: Theory and Application. Stanford University PhD thesis.
de Farias, D. P., Van Roy, B. (2003). The linear programming approach to approximate dynamic
programming. Operations Research 51 (6):850–865.
Dean, T., Lin, S.-H. (1995). Decomposition techniques for planning in stochastic domains.
In Proceedings of the Fourteenth International Joint Conference on Artificial Intelligence,
pp. 1121–1127. Morgan Kaufmann. See also Technical Report CS-95-10, Brown University,
Department of Computer Science, 1995.
Degris, T., Pilarski, P. M., Sutton, R. S. (2012). Model-free reinforcement learning with
continuous action in practice. In 2012 American Control Conference, pp. 2177–2182. IEEE.
Degris, T., White, M., Sutton, R. S. (2012). O↵-policy actor–critic. In Proceedings of the 29th
International Conference on Machine Learning. ArXiv:1205.4839, 2012.
Denardo, E. V. (1967). Contraction mappings in the theory underlying dynamic programming.
SIAM Review, 9(2):165–177.
Dennett, D. C. (1978). Why the Law of E↵ect Will Not Go Away. Brainstorms, pp. 71–89.
Bradford/MIT Press, Cambridge, MA.
Derthick, M. (1984). Variations on the Boltzmann machine learning algorithm. Carnegie-Mellon
University Department of Computer Science Technical Report No. CMU-CS-84-120.
Deutsch, J. A. (1953). A new type of behaviour theory. British Journal of Psychology. General
Section, 44(4):304–317.

490

References

Deutsch, J. A. (1954). A machine with insight. Quarterly Journal of Experimental Psychology,
6(1):6–11.
Dick, T. (2015). Policy Gradient Reinforcement Learning Without Regret. M.Sc. thesis,
University of Alberta.
Dickinson, A. (1980). Contemporary Animal Learning Theory. Cambridge University Press.
Dickinson, A. (1985). Actions and habits: the development of behavioral autonomy. Phil. Trans.
R. Soc. Lond. B, 308(1135):67–78.
Dickinson, A., Balleine, B. W. (2002). The role of learning in motivation. In C. R. Gallistel(Ed.), Stevens’ Handbook of Experimental Psychology, volume 3, pp. 497–533. Wiley, NY.
Dietterich, T. G., Buchanan, B. G. (1984). The role of the critic in learning systems. In O. G.
Selfridge, E. L. Rissland, and M. A. Arbib (Eds.), Adaptive Control of Ill-Defined Systems,
pp. 127–147. Plenum Press, NY.
Dietterich, T. G., Flann, N. S. (1995). Explanation-based learning and reinforcement learning:
A unified view. In A. Prieditis and S. Russell (Eds.), Proceedings of the 12th International
Conference on Machine Learning, pp. 176–184. Morgan Kaufmann.
Dietterich, T. G., Wang, X. (2002). Batch value function approximation via support vectors.
In Advances in Neural Information Processing Systems 14, pp. 1491–1498. MIT Press,
Cambridge, MA.
Diuk, C., Cohen, A., Littman, M. L. (2008). An object-oriented representation for efficient
reinforcement learning. In Proceedings of the 25th International Conference on Machine
Learning, pp. 240–247. ACM, New York.
Dolan, R. J., Dayan, P. (2013). Goals and habits in the brain. Neuron, 80(2):312–325.
Doll, B. B., Simon, D. A., Daw, N. D. (2012). The ubiquity of model-based reinforcement
learning. Current Opinion in Neurobiology, 22(6):1–7.
Donahoe, J. W., Burgos, J. E. (2000). Behavior analysis and revaluation. Journal of theExperimental Analysis of Behavior, 74(3):331–346.
Dorigo, M., Colombetti, M. (1994). Robot shaping: Developing autonomous agents throughlearning. Artificial Intelligence, 71(2):321–370.
Doya, K. (1996). Temporal di↵erence learning in continuous time and space. In Advances in
Neural Information Processing Systems 8, pp. 1073–1079. MIT Press, Cambridge, MA.
Doya, K., Sejnowski, T. J. (1995). A novel reinforcement model of birdsong vocalization
learning. In Advances in Neural Information Processing Systems 7, pp. 101–108. MIT Press,
Cambridge, MA.
Doya, K., Sejnowski, T. J. (1998). A computational model of birdsong learning by auditory
experience and auditory feedback. In P. W. F. Poon and J. F. Brugge (Eds.), Central
Auditory Processing and Neural Modeling, pp. 77–88. Springer, Boston, MA.
Doyle, P. G., Snell, J. L. (1984). Random Walks and Electric Networks. The Mathematical
Association of America. Carus Mathematical Monograph 22.
Dreyfus, S. E., Law, A. M. (1977). The Art and Theory of Dynamic Programming. Academic
Press, New York.
Du, S. S., Chen, J., Li, L., Xiao, L., Zhou, D. (2017). Stochastic variance reduction methods forpolicy evaluation. Proceedings of the 34th International Conference on Machine Learning,pp. 1049–1058. ArXiv:1702.07944.
Duda, R. O., Hart, P. E. (1973). Pattern Classification and Scene Analysis. Wiley, New York.

References

491

Du↵, M. O. (1995). Q-learning for bandit problems. In Proceedings of the 12th International
Conference on Machine Learning, pp. 209–217. Morgan Kaufmann.
Egger, D. M., Miller, N. E. (1962). Secondary reinforcement in rats as a function of information
value and reliability of the stimulus. Journal of Experimental Psychology, 64:97–104.
Eshel, N., Tian, J., Bukwich, M., Uchida, N. (2016). Dopamine neurons share common response
function for reward prediction error. Nature Neuroscience, 19 (3):479–486.
Estes, W. K. (1943). Discriminative conditioning. I. A discriminative property of conditioned
anticipation. Journal of Experimental Psychology, 32 (2):150–155.
Estes, W. K. (1948). Discriminative conditioning. II. E↵ects of a Pavlovian conditioned stimulus
upon a subsequently established operant response. Journal of Experimental Psychology,
38 (2):173–177.
Estes, W. K. (1950). Toward a statistical theory of learning. Psychololgical Review, 57(2):94–107.
Farley, B. G., Clark, W. A. (1954). Simulation of self-organizing systems by digital computer.
IRE Transactions on Information Theory, 4(4):76–84.
Farries, M. A., Fairhall, A. L. (2007). Reinforcement learning with modulated spike timingdependent synaptic plasticity. Journal of Neurophysiology, 98(6):3648–3665.
Feldbaum, A. A. (1965). Optimal Control Systems. Academic Press, New York.
Finch, G., Culler, E. (1934). Higher order conditioning with constant motivation. The American
Journal of Psychology:596–602.
Finnsson, H., Björnsson, Y. (2008). Simulation-based approach to general game playing. In
Proceedings of the Association for the Advancement of Artificial Intelligence, pp. 259–264.
Fiorillo, C. D., Yun, S. R., Song, M. R. (2013). Diversity and homogeneity in responses of
midbrain dopamine neurons. The Journal of Neuroscience, 33(11):4693–4709.
Florian, R. V. (2007). Reinforcement learning through modulation of spike-timing-dependent
synaptic plasticity. Neural Computation, 19(6):1468–1502.
Fogel, L. J., Owens, A. J., Walsh, M. J. (1966). Artificial Intelligence through Simulated Evolution.
John Wiley and Sons.
French, R. M. (1999). Catastrophic forgetting in connectionist networks. Trends in cognitive
sciences, 3 (4):128–135.
Frey, U., Morris, R. G. M. (1997). Synaptic tagging and long-term potentiation. Nature,385(6616):533–536.
Frémaux, N., Sprekeler, H., Gerstner, W. (2010). Functional requirements for reward-modulated
spike-timing-dependent plasticity. The Journal of Neuroscience, 30 (40): 13326–13337
Friedman, J. H., Bentley, J. L., Finkel, R. A. (1977). An algorithm for finding best matches inlogarithmic expected time. ACM Transactions on Mathematical Software, 3 (3):209–226.
Friston, K. J., Tononi, G., Reeke, G. N., Sporns, O., Edelman, G. M. (1994). Value-dependent
selection in the brain: Simulation in a synthetic neural model. Neuroscience, 59(2):229–243.
Fu, K. S. (1970). Learning control systems—Review and outlook. IEEE Transactions on
Automatic Control, 15(2):210–221.
Galanter, E., Gerstenhaber, M. (1956). On thought: The extrinsic theory. Psychological Review,
63(4):218–227.
Gallistel, C. R. (2005). Deconstructing the law of e↵ect. Games and Economic Behavior,
52 (2):410–423.
Gardner, M. (1973). Mathematical games. Scientific American, 228(1):108–115.
Geist, M., Scherrer, B. (2014). O↵-policy learning with eligibility traces: A survey. Journal of
Machine Learning Research, 15 (1):289–333.

492

References

Gelly, S., Silver, D. (2007). Combining online and o✏ine knowledge in UCT. Proceedings of the24th International Conference on Machine Learning, pp. 273–280.
Gelperin, A., Hopfield, J. J., Tank, D. W. (1985). The logic of limax learning. In A. Selverston(Ed.), Model Neural Networks and Behavior, pp. 247–261. Plenum Press, New York.
Genesereth, M., Thielscher, M. (2014). General game playing. Synthesis Lectures on Artificial
Intelligence and Machine Learning, 8 (2):1–229.
Gershman, S. J., Moustafa, A. A., Ludvig, E. A. (2014). Time representation in reinforcement
learning models of the basal ganglia. Frontiers in Computational Neuroscience, 7:194.
Gershman, S. J., Pesaran, B., Daw, N. D. (2009). Human reinforcement learning subdivides
structured action spaces by learning e↵ector-specific values. The Journal of Neuroscience,
29 (43):13524–13531.
Ghiassian, S., Rafiee, B., Sutton, R. S. (2016). A first empirical study of emphatic temporaldi↵erence learning. Workshop on Continual Learning and Deep Learning at the Conference
on Neural Information Processing Systems. ArXiv:1705.04185.
Ghiassian, S., Patterson, A., White, M., Sutton, R. S., White, A. (2018). Online o↵-policy
prediction. ArXiv:1811.02597.
Gibbs, C. M., Cool, V., Land, T., Kehoe, E. J., Gormezano, I. (1991). Second-order conditioningof the rabbit’s nictitating membrane response. Integrative Physiological and Behavioral
Science, 26 (4):282–295.
Gittins, J. C., Jones, D. M. (1974). A dynamic allocation index for the sequential design ofexperiments. In J. Gani, K. Sarkadi, and I. Vincze (Eds.), Progress in Statistics, pp. 241–266.
North-Holland, Amsterdam–London.
Glimcher, P. W. (2011). Understanding dopamine and reinforcement learning: The dopamine
reward prediction error hypothesis. Proceedings of the National Academy of Sciences,
108(Supplement 3):15647–15654.
Glimcher, P. W. (2003). Decisions, Uncertainty, and the Brain: The science of Neuroeconomics.
MIT Press, Cambridge, MA.
Glimcher, P. W., Fehr, E. (Eds.) (2013). Neuroeconomics: Decision Making and the Brain,
Second Edition. Academic Press.
Goethe, J. W. V. (1878). The Sorcerer’s Apprentice. In The Permanent Goethe, p. 349. The
Dial Press, Inc., New York.
Goldstein, H. (1957). Classical Mechanics. Addison-Wesley, Reading, MA.
Goodfellow, I., Bengio, Y., Courville, A. (2016). Deep Learning. MIT Press, Cambridge, MA.
Goodwin, G. C., Sin, K. S. (1984). Adaptive Filtering Prediction and Control. Prentice-Hall,Englewood Cli↵s, NJ.
Gopnik, A., Glymour, C., Sobel, D., Schulz, L. E., Kushnir, T., Danks, D. (2004). A theory of
causal learning in children: Causal maps and Bayes nets. Psychological Review, 111(1):3–32.
Gordon, G. J. (1995). Stable function approximation in dynamic programming. In A. Prieditis
and S. Russell (Eds.), Proceedings of the 12th International Conference on Machine Learning,pp. 261–268. Morgan Kaufmann. An expanded version was published as Technical Report
CMU-CS-95-103. Carnegie Mellon University, Pittsburgh, PA, 1995.
Gordon, G. J. (1996a). Chattering in SARSA( ). CMU learning lab internal report.
Gordon, G. J. (1996b). Stable fitted reinforcement learning. In Advances in Neural Information
Processing Systems 8, pp. 1052–1058. MIT Press, Cambridge, MA.
Gordon, G. J. (1999). Approximate Solutions to Markov Decision Processes. PhD thesis,
Carnegie Mellon University, Pittsburgh PA. Pittsburgh, PA.
Gordon, G. J. (2001). Reinforcement learning with function approximation converges to a

References493

region. In Advances in Neural Information Processing Systems 13, pp. 1040–1046. MIT
Press, Cambridge, MA.
Graybiel, A. M. (2000). The basal ganglia. Current Biology, 10(14):R509–R511.
Greensmith, E., Bartlett, P. L., Baxter, J. (2002). Variance reduction techniques for gradientestimates in reinforcement learning. In Advances in Neural Information Processing Systems
14, pp. 1507–1514. MIT Press, Cambridge, MA.
Greensmith, E., Bartlett, P. L., Baxter, J. (2004). Variance reduction techniques for gradientestimates in reinforcement learning. Journal of Machine Learning Research, 5 (Nov):1471–
1530.
Griffith, A. K. (1966). A new machine learning technique applied to the game of checkers.
Technical Report Project MAC, Artificial Intelligence Memo 94. Massachusetts Institute ofTechnology, Cambridge, MA.
Griffith, A. K. (1974). A comparison and evaluation of three machine learning procedures asapplied to the game of checkers. Artificial Intelligence, 5(2):137–148.
Grondman, I., Busoniu, L., Lopes, G. A., Babuska, R. (2012). A survey of actor–critic reinforcement learning: Standard and natural policy gradients. IEEE Transactions on Systems, Man,
and Cybernetics, Part C (Applications and Reviews), 42 (6):1291–1307.
Grossberg, S. (1975). A neural model of attention, reinforcement, and discrimination learning.
International Review of Neurobiology, 18:263–327.
Grossberg, S., Schmajuk, N. A. (1989). Neural dynamics of adaptive timing and temporal
discrimination during associative learning. Neural Networks, 2(2):79–102.
Gullapalli, V. (1990). A stochastic reinforcement algorithm for learning real-valued functions.
Neural Networks, 3(6): 671–692.
Gullapalli, V., Barto, A. G. (1992). Shaping as a method for accelerating reinforcement learning.
In Proceedings of the 1992 IEEE International Symposium on Intelligent Control, pp. 554–559.
IEEE.
Gurvits, L., Lin, L.-J., Hanson, S. J. (1994). Incremental learning of evaluation functions
for absorbing Markov chains: New methods and theorems. Siemans Corporate Research,
Princeton, NJ.
Hackman, L. (2012). Faster Gradient-TD Algorithms. M.Sc. thesis, University of Alberta,
Edmonton.
Hallak, A., Tamar, A., Mannor, S. (2015). Emphatic TD Bellman operator is a contraction.
ArXiv:1508.03411.
Hallak, A., Tamar, A., Munos, R., Mannor, S. (2016). Generalized emphatic temporal di↵erence
learning: Bias-variance analysis. In Proceedings of the Thirtieth AAAI Conference on
Artificial Intelligence, pp. 1631–1637. AAAI Press, Menlo Park, CA.
Hammer, M. (1997). The neural basis of associative reward learning in honeybees. Trends in
Neuroscience, 20(6):245–252.
Hammer, M., Menzel, R. (1995). Learning and memory in the honeybee. The Journal of
Neuroscience, 15(3):1617–1630.
Hampson, S. E. (1983). A Neural Model of Adaptive Behavior. PhD thesis, University ofCalifornia, Irvine.
Hampson, S. E. (1989). Connectionist Problem Solving: Computational Aspects of Biological
Learning. Birkhauser, Boston.
Hare, T. A., O’Doherty, J., Camerer, C. F., Schultz, W., Rangel, A. (2008). Dissociating the roleof the orbitofrontal cortex and the striatum in the computation of goal values and prediction
errors. The Journal of Neuroscience, 28(22):5623–5630.

494

References

Harth, E., Tzanakou, E. (1974). Alopex: A stochastic method for determining visual receptive
fields. Vision Research, 14 (12):1475–1482.
Hassabis, D., Maguire, E. A. (2007). Deconstructing episodic memory with construction. Trends
in Cognitive Sciences, 11(7):299–306.
Hauskrecht, M., Meuleau, N., Kaelbling, L. P., Dean, T., Boutilier, C. (1998). Hierarchicalsolution of Markov decision processes using macro-actions. In Proceedings of the Fourteenth
Conference on Uncertainty in Artificial Intelligence, pp. 220–229. Morgan Kaufmann.
Hawkins, R. D., Kandel, E. R. (1984). Is there a cell-biological alphabet for simple forms oflearning? Psychological Review, 91(3):375–391.
Haykin, S. (1994). Neural networks: A Comprehensive Foundation, Macmillan, New York.
He, K., Huertas, M., Hong, S. Z., Tie, X., Hell, J. W., Shouval, H., Kirkwood, A. (2015). Distinct
eligibility traces for LTP and LTD in cortical synapses. Neuron, 88(3):528–538.
He, K., Zhang, X., Ren, S., Sun, J. (2016). Deep residual learning for image recognition. In
Proceedings of the 1992 IEEE Conference on Computer Vision and Pattern Recognition,
pp. 770–778.
Hebb, D. O. (1949). The Organization of Behavior: A Neuropsychological Theory. John Wileyand Sons Inc., New York. Reissued by Lawrence Erlbaum Associates Inc., Mahwah NJ, 2002.
Hengst, B. (2012). Hierarchical approaches. In M. Wiering and M. van Otterlo (Eds.), Reinforcement Learning: State-of-the-Art, pp. 293–323. Springer-Verlag Berlin Heidelberg.
Herrnstein, R. J. (1970). On the Law of E↵ect. Journal of the Experimental Analysis of Behavior,
13 (2):243–266.
Hersh, R., Griego, R. J. (1969). Brownian motion and potential theory. Scientific American,
220(3):66–74.
Hester, T., Stone, P. (2012). Learning and using models. In M. Wiering and M. van Otterlo (Eds.),
Reinforcement Learning: State-of-the-Art, pp. 111–141. Springer-Verlag Berlin Heidelberg.
Hesterberg, T. C. (1988), Advances in Importance Sampling, PhD thesis, Statistics Department,
Stanford University.
Hilgard, E. R. (1956). Theories of Learning, Second Edition. Appleton-Century-Cofts, Inc.,New York.
Hilgard, E. R., Bower, G. H. (1975). Theories of Learning. Prentice-Hall, Englewood Cli↵s, NJ.
Hinton, G. E. (1984). Distributed representations. Technical Report CMU-CS-84-157. Department of Computer Science, Carnegie-Mellon University, Pittsburgh, PA.
Hinton, G. E., Osindero, S., Teh, Y. (2006). A fast learning algorithm for deep belief nets.
Neural Computation, 18(7):1527–1554.
Hochreiter, S., Schmidhuber, J. (1997). LTSM can solve hard time lag problems. In Advancesin Neural Information Processing Systems 9, pp. 473–479. MIT Press, Cambridge, MA.
Holland, J. H. (1975). Adaptation in Natural and Artificial Systems. University of Michigan
Press, Ann Arbor.
Holland, J. H. (1976). Adaptation. In R. Rosen and F. M. Snell (Eds.), Progress in Theoretical
Biology, vol. 4, pp. 263–293. Academic Press, New York.
Holland, J. H. (1986). Escaping brittleness: The possibility of general-purpose learning algorithms
applied to rule-based systems. In R. S. Michalski, J. G. Carbonell, and T. M. Mitchell(Eds.), Machine Learning: An Artificial Intelligence Approach, vol. 2, pp. 593–623. Morgan
Kaufmann.
Hollerman, J. R., Schultz, W. (1998). Dopmine neurons report an error in the temporal
prediction of reward during learning. Nature Neuroscience, 1(4):304–309.

References

495

Houk, J. C., Adams, J. L., Barto, A. G. (1995). A model of how the basal ganglia generates anduses neural signals that predict reinforcement. In J. C. Houk, J. L. Davis, and D. G. Beiser(Eds.), Models of Information Processing in the Basal Ganglia, pp. 249–270. MIT Press,
Cambridge, MA.
Howard, R. (1960). Dynamic Programming and Markov Processes. MIT Press, Cambridge, MA.
Hull, C. L. (1932). The goal-gradient hypothesis and maze learning. Psychological Review,
39(1):25–43.
Hull, C. L. (1943). Principles of Behavior. Appleton-Century, New York.
Hull, C. L. (1952). A Behavior System. Wiley, New York.
Io↵e, S., Szegedy, C. (2015). Batch normalization: Accelerating deep network training by
reducing internal covariate shift. ArXiv:1502.03167.
İpek, E., Mutlu, O., Martı́nez, J. F., Caruana, R. (2008). Self-optimizing memory controllers: A
reinforcement learning approach. In ISCA’08:Proceedings of the 35th Annual International
Symposium on Computer Architecture, pp. 39–50. IEEE Computer Society Washington, DC.
Izhikevich, E. M. (2007). Solving the distal reward problem through linkage of STDP anddopamine signaling. Cerebral Cortex, 17(10):2443–2452.
Jaakkola, T., Jordan, M. I., Singh, S. P. (1994). On the convergence of stochastic iterative
dynamic programming algorithms. Neural Computation, 6:1185–1201.
Jaakkola, T., Singh, S. P., Jordan, M. I. (1995). Reinforcement learning algorithm for partiallyobservable Markov decision problems. In Advances in Neural Information Processing Systems
7, pp. 345–352. MIT Press, Cambridge, MA.
Jacobs, R. A. (1988). Increased rates of convergence through learning rate adaptation. Neural
Networks, 1 (4):295–307.
Jaderberg, M., Mnih, V., Czarnecki, W. M., Schaul, T., Leibo, J. Z., Silver, D., Kavukcuoglu, K.
(2016). Reinforcement learning with unsupervised auxiliary tasks. ArXiv:1611.05397.
Jaeger, H. (1997). Observable operator models and conditioned continuation representations. Arbeitspapiere der GMD 1043, GMD Forschungszentrum Informationstechnik, Sankt Augustin,
Germany.
Jaeger, H. (1998). Discrete Time, Discrete Valued Observable Operator Models: A Tutorial.
GMD-Forschungszentrum Informationstechnik.
Jaeger, H. (2000). Observable operator models for discrete stochastic time series. NeuralComputation, 12 (6):1371–1398.
Jaeger, H. (2002). Tutorial on training recurrent neural networks, covering BPPT, RTRL, EKF
and the ‘echo state network’ approach. German National Research Center for Information
Technology, Technical Report GMD report 159, 2002.
Joel, D., Niv, Y., Ruppin, E. (2002). Actor–critic models of the basal ganglia: New anatomical
and computational perspectives. Neural Networks, 15(4):535–547.
Johnson, A., Redish, A. D. (2007). Neural ensembles in CA3 transiently encode paths forwardof the animal at a decision point. The Journal of Neuroscience, 27(45):12176–12189.
Kaelbling, L. P. (1993a). Hierarchical learning in stochastic domains: Preliminary results. In
Proceedings of the 10th International Conference on Machine Learning, pp. 167–173. Morgan
Kaufmann.
Kaelbling, L. P. (1993b). Learning in Embedded Systems. MIT Press, Cambridge, MA.
Kaelbling, L. P. (Ed.) (1996). Special triple issue on reinforcement learning, Machine Learning,22(1/2/3).
Kaelbling, L. P., Littman, M. L., Moore, A. W. (1996). Reinforcement learning: A survey.
Journal of Artificial Intelligence Research, 4:237–285.

496

References

Kakade, S. M. (2002). A natural policy gradient. In Advances in Neural Information Processing
Systems 14, pp. 1531–1538. MIT Press, Cambridge, MA.
Kakade, S. M. (2003). On the Sample Complexity of Reinforcement Learning. PhD thesis,
University of London.
Kakutani, S. (1945). Markov processes and the Dirichlet problem. Proceedings of the Japan
Academy, 21(3-10):227–233.
Kalos, M. H., Whitlock, P. A. (1986). Monte Carlo Methods. Wiley, New York.
Kamin, L. J. (1968). “Attention-like” processes in classical conditioning. In M. R. Jones (Ed.),
Miami Symposium on the Prediction of Behavior, 1967: Aversive Stimulation, pp. 9–31.
University of Miami Press, Coral Gables, Florida.
Kamin, L. J. (1969). Predictability, surprise, attention, and conditioning. In B. A. Campbell andR. M. Church (Eds.), Punishment and Aversive Behavior, pp. 279–296. Appleton-CenturyCrofts, New York.
Kandel, E. R., Schwartz, J. H., Jessell, T. M., Siegelbaum, S. A., Hudspeth, A. J. (Eds.) (2013).
Principles of Neural Science, Fifth Edition. McGraw-Hill Companies, Inc.
Karampatziakis, N., Langford, J. (2010). Online importance weight aware updates. ArXiv:1011.1576.
Kashyap, R. L., Blaydon, C. C., Fu, K. S. (1970). Stochastic approximation. In J. M. Mendel
and K. S. Fu (Eds.), Adaptive, Learning, and Pattern Recognition Systems: Theory andApplications, pp. 329–355. Academic Press, New York.
Kearney, A., Veeriah, V, Travnik, J, Sutton, R. S., Pilarski, P. M. (in preparation). TIDBD:
Adapting Temporal-di↵erence Step-sizes Through Stochastic Meta-descent.
Kearns, M., Singh, S. (2002). Near-optimal reinforcement learning in polynomial time. Machine
Learning, 49 (2-3):209–232.
Keerthi, S. S., Ravindran, B. (1997). Reinforcement learning. In E. Fieslerm and R. Beale(Eds.), Handbook of Neural Computation, C3. Oxford University Press, New York.
Kehoe, E. J. (1982). Conditioning with serial compound stimuli: Theoretical and empiricalissues. Experimental Animal Behavior, 1:30–65.
Kehoe, E. J., Schreurs, B. G., Graham, P. (1987). Temporal primacy overrides prior training
in serial compound conditioning of the rabbit’s nictitating membrane response. Animal
Learning & Behavior, 15(4):455–464.
Keiflin, R., Janak, P. H. (2015). Dopamine prediction errors in reward learning and addiction:
From theory to neural circuitry. Neuron, 88(2):247– 263.
Kimble, G. A. (1961). Hilgard and Marquis’ Conditioning and Learning. Appleton-CenturyCrofts, New York.
Kimble, G. A. (1967). Foundations of Conditioning and Learning. Appleton-Century-Crofts,
New York.
Kingma, D., Ba, J. (2014). Adam: A method for stochastic optimization. ArXiv:1412.6980.
Klopf, A. H. (1972). Brain function and adaptive systems—A heterostatic theory. Technical
Report AFCRL-72-0164, Air Force Cambridge Research Laboratories, Bedford, MA. A
summary appears in Proceedings of the International Conference on Systems, Man, andCybernetics (1974). IEEE Systems, Man, and Cybernetics Society, Dallas, TX.
Klopf, A. H. (1975). A comparison of natural and artificial intelligence. SIGART Newsletter,
53:11–13.
Klopf, A. H. (1982). The Hedonistic Neuron: A Theory of Memory, Learning, and Intelligence.
Hemisphere, Washington, DC.
Klopf, A. H. (1988). A neuronal model of classical conditioning. Psychobiology, 16(2):85–125.

References

497

Klyubin, A. S., Polani, D., Nehaniv, C. L. (2005). Empowerment: A universal agent-centricmeasure of control. In Proceedings of the 2005 IEEE Congress on Evolutionary Computation(Vol. 1, pp. 128–135). IEEE.
Kober, J., Peters, J. (2012). Reinforcement learning in robotics: A survey. In M. Wiering, M.
van Otterlo (Eds.), Reinforcement Learning: State-of-the-Art, pp. 579–610. Springer-Verlag.
Kocsis, L., Szepesvári, Cs. (2006). Bandit based Monte-Carlo planning. In Proceedings of theEuropean Conference on Machine Learning, pp. 282–293. Springer-Verlag Berlin Heidelberg.
Kohonen, T. (1977). Associative Memory: A System Theoretic Approach. Springer-Verlag,
Berlin.
Koller, D., Friedman, N. (2009). Probabilistic Graphical Models: Principles and Techniques.
MIT Press.
Kolodziejski, C., Porr, B., Wörgötter, F. (2009). On the asymptotic equivalence betweendi↵erential Hebbian and temporal di↵erence learning. Neural Computation, 21(4):1173–1202.
Kolter, J. Z. (2011). The fixed points of o↵-policy TD. In Advances in Neural Information
Processing Systems 24, pp. 2169–2177. Curran Associates, Inc.
Konda, V. R., Tsitsiklis, J. N. (2000). Actor-critic algorithms. In Advances in Neural Information
Processing Systems 12, pp. 1008–1014. MIT Press, Cambridge, MA.
Konda, V. R., Tsitsiklis, J. N. (2003). On actor-critic algorithms. SIAM Journal on Control
and Optimization, 42 (4):1143–1166.
Konidaris, G. D., Osentoski, S., Thomas, P. S. (2011). Value function approximation in
reinforcement learning using the Fourier basis . In Proceedings of the Twenty-Fifth Conferenceof the Association for the Advancement of Artificial Intelligence, pp. 380–385.
Korf, R. E. (1988). Optimal path finding algorithms. In L. N. Kanal and V. Kumar (Eds.),
Search in Artificial Intelligence, pp. 223–267. Springer-Verlag, Berlin.
Korf, R. E. (1990). Real-time heuristic search. Artificial Intelligence, 42 (2–3), 189–211.
Koshland, D. E. (1980). Bacterial Chemotaxis as a Model Behavioral System. Raven Press, NewYork.
Koza, J. R. (1992). Genetic Programming: On the Programming of Computers by Means of
Natural Selection (Vol. 1). MIT Press., Cambridge, MA.
Kraft, L. G., Campagna, D. P. (1990). A summary comparison of CMAC neural network andtraditional adaptive control systems. In T. Miller, R. S. Sutton, and P. J. Werbos (Eds.),
Neural Networks for Control, pp. 143–169. MIT Press, Cambridge, MA.
Kraft, L. G., Miller, W. T., Dietz, D. (1992). Development and application of CMAC neuralnetwork-based control. In D. A. White and D. A. Sofge (Eds.), Handbook of Intelligent
Control: Neural, Fuzzy, and Adaptive Approaches, pp. 215–232. Van Nostrand Reinhold,
New York.
Kumar, P. R., Varaiya, P. (1986). Stochastic Systems: Estimation, Identification, and Adaptive
Control. Prentice-Hall, Englewood Cli↵s, NJ.
Kumar, P. R. (1985). A survey of some results in stochastic adaptive control. SIAM Journal ofControl and Optimization, 23(3):329–380.
Kumar, V., Kanal, L. N. (1988). The CDP, A unifying formulation for heuristic search, dynamic
programming, and branch-and-bound. In L. N. Kanal and V. Kumar (Eds.), Search in
Artificial Intelligence, pp. 1–37. Springer-Verlag, Berlin.
Kushner, H. J., Dupuis, P. (1992). Numerical Methods for Stochastic Control Problems in
Continuous Time. Springer-Verlag, New York.

498

References

Kuvayev, L., Sutton, R.S. (1996). Model-based reinforcement learning with an approximate,learned model. Proceedings of the Ninth Yale Workshop on Adaptive and Learning Systems,
pp. 101–105, Yale University, New Haven, CT.
Lagoudakis, M., Parr, R. (2003). Least squares policy iteration. Journal of Machine Learning
Research, 4 (Dec):1107–1149.
Lai, T. L., Robbins, H. (1985). Asymptotically efficient adaptive allocation rules. Advances inApplied Mathematics, 6 (1):4–22.
Lakshmivarahan, S., Narendra, K. S. (1982). Learning algorithms for two-person zero-sum
stochastic games with incomplete information: A unified approach. SIAM Journal of Control
and Optimization, 20(4):541–552.
Lammel, S., Lim, B. K., Malenka, R. C. (2014). Reward and aversion in a heterogeneous
midbrain dopamine system. Neuropharmacology, 76:353–359.
Lane, S. H., Handelman, D. A., Gelfand, J. J. (1992). Theory and development of higher-order
CMAC neural networks. IEEE Control Systems, 12 (2):23–30.
LeCun, Y. (1985). Une procédure d’apprentissage pour réseau a seuil asymmetrique (a learning
scheme for asymmetric threshold networks). In Proceedings of Cognitiva 85, Paris, France.
LeCun, Y., Bottou, L., Bengio, Y., Ha↵ner, P. (1998). Gradient-based learning applied to
document recognition. Proceedings of the IEEE, 86(11):2278–2324.
Legenstein, R. W., Maass, D. P. (2008). A learning theory for reward-modulated spike-timingdependent plasticity with application to biofeedback. PLoS Computational Biology, 4(10).
Levy, W. B., Steward, D. (1983). Temporal contiguity requirements for long-term associative
potentiation/depression in the hippocampus. Neuroscience, 8(4):791–797.
Lewis, F. L., Liu, D. (Eds.) (2012). Reinforcement Learning and Approximate Dynamic
Programming for Feedback Control. John Wiley and Sons.
Lewis, R. L., Howes, A., Singh, S. (2014). Computational rationality: Linking mechanism andbehavior through utility maximization. Topics in Cognitive Science, 6(2):279–311.
Li, L. (2012). Sample complexity bounds of exploration. In M. Wiering and M. van Otterlo (Eds.),
Reinforcement Learning: State-of-the-Art, pp. 175–204. Springer-Verlag Berlin Heidelberg.
Li, L., Chu, W., Langford, J., Schapire, R. E. (2010). A contextual-bandit approach to personalized news article recommendation. In Proceedings of the 19th International Conference on
World Wide Web, pp. 661–670. ACM, New York.
Lin, C.-S., Kim, H. (1991). CMAC-based adaptive critic self-learning control. IEEE Transactions
on Neural Networks, 2(5):530–533.
Lin, L.-J. (1992). Self-improving reactive agents based on reinforcement learning, planning andteaching. Machine Learning, 8(3-4):293–321.
Lin, L.-J., Mitchell, T. (1992). Reinforcement learning with hidden states. In Proceedings of
the Second International Conference on Simulation of Adaptive Behavior: From Animals to
Animats, pp. 271–280. MIT Press, Cambridge, MA.
Littman, M. L., Cassandra, A. R., Kaelbling, L. P. (1995). Learning policies for partiallyobservable environments: Scaling up. In Proceedings of the 12th International Conference on
Machine Learning, pp. 362–370. Morgan Kaufmann.
Littman, M. L., Dean, T. L., Kaelbling, L. P. (1995). On the complexity of solving Markov
decision problems. In Proceedings of the Eleventh Annual Conference on Uncertainty in
Artificial Intelligence, pp. 394–402.
Littman, M. L., Sutton, R. S., Singh, S. (2002). Predictive representations of state. In Advancesin Neural Information Processing Systems 14, pp. 1555–1561. MIT Press, Cambridge, MA.
Liu, J. S. (2001). Monte Carlo Strategies in Scientific Computing. Springer-Verlag, Berlin.

References

499

Ljung, L. (1998). System identification. In A. Procházka, J. Uhlı́r̂, P. W. J. Rayner, and N. G.
Kingsbury (Eds.), Signal Analysis and Prediction, pp. 163–173. Springer Science + BusinessMedia New York, LLC.
Ljung, L., Söderstrom, T. (1983). Theory and Practice of Recursive Identification. MIT Press,
Cambridge, MA.
Ljungberg, T., Apicella, P., Schultz, W. (1992). Responses of monkey dopamine neurons during
learning of behavioral reactions. Journal of Neurophysiology, 67(1):145–163.
Lovejoy, W. S. (1991). A survey of algorithmic methods for partially observed Markov decision
processes. Annals of Operations Research, 28(1):47–66.
Luce, D. (1959). Individual Choice Behavior. Wiley, New York.
Ludvig, E. A., Bellemare, M. G., Pearson, K. G. (2011). A primer on reinforcement learningin the brain: Psychological, computational, and neural perspectives. In E. Alonso and E.
Mondragón (Eds.), Computational Neuroscience for Advancing Artificial Intelligence: Models,Methods and Applications, pp. 111–44. Medical Information Science Reference, Hershey PA.
Ludvig, E. A., Sutton, R. S., Kehoe, E. J. (2008). Stimulus representation and the timing of
reward-prediction errors in models of the dopamine system. Neural Computation, 20(12):3034–
3054.
Ludvig, E. A., Sutton, R. S., Kehoe, E. J. (2012). Evaluating the TD model of classicalconditioning. Learning & behavior, 40(3):305–319.
Machado, A. (1997). Learning the temporal dynamics of behavior. Psychological Review,104(2):241–265.
Mackintosh, N. J. (1975). A theory of attention: Variations in the associability of stimuli withreinforcement. Psychological Review, 82(4):276–298.
Mackintosh, N. J. (1983). Conditioning and Associative Learning. Clarendon Press, Oxford.
Maclin, R., Shavlik, J. W. (1994). Incorporating advice into agents that learn from reinforcements.
In Proceedings of the Twelfth National Conference on Artificial Intelligence, pp. 694–699.
AAAI Press, Menlo Park, CA.
Maei, H. R. (2011). Gradient Temporal-Di↵erence Learning Algorithms. PhD thesis, Universityof Alberta, Edmonton.
Maei, H. R. (2018). Convergent actor-critic algorithms under o↵-policy training and function
approximation. ArXiv:1802.07842.
Maei, H. R., Sutton, R. S. (2010). GQ( ): A general gradient algorithm for temporal-di↵erence
prediction learning with eligibility traces. In Proceedings of the Third Conference on Artificial
General Intelligence, pp. 91–96.
Maei, H. R., Szepesvári, Cs., Bhatnagar, S., Precup, D., Silver, D., Sutton, R. S. (2009).
Convergent temporal-di↵erence learning with arbitrary smooth function approximation. In
Advances in Neural Information Processing Systems 22, pp. 1204–1212. Curran Associates,Inc.
Maei, H. R., Szepesvári, Cs., Bhatnagar, S., Sutton, R. S. (2010). Toward o↵-policy learning
control with function approximation. In Proceedings of the 27th International Conference on
Machine Learning, pp. 719–726).
Mahadevan, S. (1996). Average reward reinforcement learning: Foundations, algorithms, andempirical results. Machine Learning, 22(1):159–196.
Mahadevan, S., Liu, B., Thomas, P., Dabney, W., Giguere, S., Jacek, N., Gemp, I., Liu, J. (2014).
Proximal reinforcement learning: A new theory of sequential decision making in primal-dual
spaces. ArXiv:1405.6757.
Mahadevan, S., Connell, J. (1992). Automatic programming of behavior-based robots using

500

References

reinforcement learning. Artificial Intelligence, 55(2-3):311–365.
Mahmood, A. R. (2017). Incremental O↵-Policy Reinforcement Learning Algorithms. PhD
thesis, University of Alberta, Edmonton.
Mahmood, A. R., Sutton, R. S. (2015). O↵-policy learning based on weighted importance sampling with linear computational complexity. In Proceedings of the 31st Conference on Uncertainty in Artificial Intelligence,
pp. 552–561.
AUAI Press Corvallis,
Oregon.
Mahmood, A. R., Sutton, R. S., Degris, T., Pilarski, P. M. (2012). Tuning-free step-size adaptation. In 2012 IEEE International Conference on Acoustics, Speech and Signal Processing,Proceedings, pp. 2121–2124. IEEE.
Mahmood, A. R., Yu, H, Sutton, R. S. (2017). Multi-step o↵-policy learning without importancesampling ratios. ArXiv:1702.03006.
Mahmood, A. R., van Hasselt, H., Sutton, R. S. (2014). Weighted importance sampling foro↵-policy learning with linear function approximation. Advances in Neural Information
Processing Systems 27, pp. 3014–3022. Curran Associates, Inc.
Marbach, P., Tsitsiklis, J. N. (1998). Simulation-based optimization of Markov reward processes.
MIT Technical Report LIDS-P-2411.
Marbach, P., Tsitsiklis, J. N. (2001). Simulation-based optimization of Markov reward processes.
IEEE Transactions on Automatic Control, 46 (2):191–209.
Markram, H., Lübke, J., Frotscher, M., Sakmann, B. (1997). Regulation of synaptic efficacy bycoincidence of postsynaptic APs and EPSPs. Science, 275(5297):213–215.
Martı́nez, J. F., İpek, E. (2009). Dynamic multicore resource management: A machine learning
approach. Micro, IEEE, 29(5):8–17.
Mataric, M. J. (1994). Reward functions for accelerated learning. In Proceedings of the 11th
International Conference on Machine Learning, pp. 181–189. Morgan Kaufmann.
Matsuda, W., Furuta, T., Nakamura, K. C., Hioki, H., Fujiyama, F., Arai, R., Kaneko, T.
(2009). Single nigrostriatal dopaminergic neurons form widely spread and highly dense
axonal arborizations in the neostriatum. The Journal of Neuroscience, 29(2):444–453.
Mazur, J. E. (1994). Learning and Behavior, 3rd ed. Prentice-Hall, Englewood Cli↵s, NJ.
McCallum, A. K. (1993). Overcoming incomplete perception with utile distinction memory. In
Proceedings of the 10th International Conference on Machine Learning, pp. 190–196. Morgan
Kaufmann.
McCallum, A. K. (1995). Reinforcement Learning with Selective Perception and Hidden State.
PhD thesis, University of Rochester, Rochester NY.
McCloskey, M., Cohen, N. J. (1989). Catastrophic interference in connectionist networks: The
sequential learning problem. Psychology of Learning and Motivation, 24 :109–165.
McClure, S. M., Daw, N. D., Montague, P. R. (2003). A computational substrate for incentive
salience. Trends in Neurosciences, 26 (8):423–428.
McCulloch, W. S., Pitts, W. (1943). A logical calculus of the ideas immanent in nervous activity.
Bulletin of Mathematical Biophysics, 5 (4):115–133.
McMahan, H. B., Gordon, G. J. (2005). Fast Exact Planning in Markov Decision Processes.
In Proceedings of the International Conference on Automated Planning and Scheduling,pp. 151-160.
Melo, F. S., Meyn, S. P., Ribeiro, M. I. (2008). An analysis of reinforcement learning withfunction approximation. In Proceedings of the 25th International Conference on Machine
Learning, pp. 664–671.
Mendel, J. M. (1966). A survey of learning control systems. ISA Transactions, 5:297–303.

References

501

Mendel, J. M., McLaren, R. W. (1970). Reinforcement learning control and pattern recognition
systems. In J. M. Mendel and K. S. Fu (Eds.), Adaptive, Learning and Pattern Recognition
Systems: Theory and Applications, pp. 287–318. Academic Press, New York.
Michie, D. (1961). Trial and error. In S. A. Barnett and A. McLaren (Eds.), Science Survey,
Part 2, pp. 129–145. Penguin, Harmondsworth.
Michie, D. (1963). Experiments on the mechanisation of game learning. 1. characterization of
the model and its parameters. The Computer Journal, 6(3):232–263.
Michie, D. (1974). On Machine Intelligence. Edinburgh University Press, Edinburgh.
Michie, D., Chambers, R. A. (1968). BOXES, An experiment in adaptive control. In E. Dale
and D. Michie (Eds.), Machine Intelligence 2, pp. 137–152. Oliver and Boyd, Edinburgh.
Miller, R. (1981). Meaning and Purpose in the Intact Brain: A Philosophical, Psychological,
and Biological Account of Conscious Process. Clarendon Press, Oxford.
Miller, W. T., An, E., Glanz, F., Carter, M. (1990). The design of CMAC neural networks forcontrol. Adaptive and Learning Systems, 1 :140–145.
Miller, W. T., Glanz, F. H. (1996). UNH CMAC verison 2.1: The University of New Hampshire
Implementation of the Cerebellar Model Arithmetic Computer - CMAC. Robotics Laboratory
Technical Report, University of New Hampshire, Durham.
Miller, S., Williams, R. J. (1992). Learning to control a bioreactor using a neural net Dyna-Q
system. In Proceedings of the Seventh Yale Workshop on Adaptive and Learning Systems,
pp. 167–172. Center for Systems Science, Dunham Laboratory, Yale University, New Haven.
Miller, W. T., Scalera, S. M., Kim, A. (1994). Neural network control of dynamic balance for abiped walking robot. In Proceedings of the Eighth Yale Workshop on Adaptive and Learning
Systems, pp. 156–161. Center for Systems Science, Dunham Laboratory, Yale University,
New Haven.
Minton, S. (1990). Quantitative results concerning the utility of explanation-based learning.
Artificial Intelligence, 42 (2-3):363–391.
Minsky, M. L. (1954). Theory of Neural-Analog Reinforcement Systems and Its Application to
the Brain-Model Problem. PhD thesis, Princeton University.
Minsky, M. L. (1961). Steps toward artificial intelligence. Proceedings of the Institute of RadioEngineers, 49:8–30. Reprinted in E. A. Feigenbaum and J. Feldman (Eds.), Computers andThought, pp. 406–450. McGraw-Hill, New York, 1963.
Minsky, M. L. (1967). Computation: Finite and Infinite Machines. Prentice-Hall, EnglewoodCli↵s, NJ.
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., Riedmiller, M.
(2013). Playing atari with deep reinforcement learning. ArXiv:1312.5602.
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves,A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A.,
Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., Hassabis, D. (2015). Humanlevel control through deep reinforcement learning. Nature, 518(7540):529–533.
Modayil, J., Sutton, R. S. (2014). Prediction driven behavior: Learning predictions that drive
fixed responses. In AAAI-14 Workshop on Artificial Intelligence and Robotics, Quebec City,
Canada.
Modayil, J., White, A., Sutton, R. S. (2014). Multi-timescale nexting in a reinforcement learning
robot. Adaptive Behavior, 22(2):146–160.
Monahan, G. E. (1982). State of the art—a survey of partially observable Markov decision
processes: theory, models, and algorithms. Management Science, 28 (1):1–16.

502

References

Montague, P. R., Dayan, P., Nowlan, S. J., Pouget, A., Sejnowski, T. J. (1993). Using aperiodic
reinforcement for directed self-organization during development. In Advances in Neural
Information Processing Systems 5, pp. 969–976. Morgan Kaufmann.
Montague, P. R., Dayan, P., Person, C., Sejnowski, T. J. (1995). Bee foraging in uncertain
environments using predictive hebbian learning. Nature, 377(6551):725–728.
Montague, P. R., Dayan, P., Sejnowski, T. J. (1996). A framework for mesencephalic dopamine
systems based on predictive Hebbian learning. The Journal of Neuroscience, 16(5):1936–1947.
Montague, P. R., Dolan, R. J., Friston, K. J., Dayan, P. (2012). Computational psychiatry.
Trends in Cognitive Sciences, 16 (1):72–80.
Montague, P. R., Sejnowski, T. J. (1994). The predictive brain: Temporal coincidence andtemporal order in synaptic learningmechanisms. Learning & Memory, 1(1):1–33.
Moore, A. W. (1990). Efficient Memory-Based Learning for Robot Control. PhD thesis,
University of Cambridge.
Moore, A. W., Atkeson, C. G. (1993). Prioritized sweeping: Reinforcement learning with lessdata and less real time. Machine Learning, 13(1):103–130.
Moore, A. W., Schneider, J., Deng, K. (1997). Efficient locally weighted polynomial regression
predictions. In Proceedings of the 14th International Conference on Machine Learning.
Morgan Kaufmann.
Moore, J. W., Blazis, D. E. J. (1989). Simulation of a classically conditioned response: A
cerebellar implementation of the sutton-barto-desmond model. In J. H. Byrne and W. O.
Berry (Eds.), Neural Models of Plasticity, pp. 187–207. Academic Press, San Diego, CA.
Moore, J. W., Choi, J.-S., Brunzell, D. H. (1998). Predictive timing under temporal uncertainty:
The time derivative model of the conditioned response. In D. A. Rosenbaum and C. E.
Collyer (Eds.), Timing of Behavior, pp. 3–34. MIT Press, Cambridge, MA.
Moore, J. W., Desmond, J. E., Berthier, N. E., Blazis, E. J., Sutton, R. S., Barto, A. G. (1986).
Simulation of the classically conditioned nictitating membrane response by a neuron-like
adaptive element: I. Response topography, neuronal firing, and interstimulus intervals.
Behavioural Brain Research, 21(2):143–154.
Moore, J. W., Marks, J. S., Castagna, V. E., Polewan, R. J. (2001). Parameter stability in the
TD model of complex CR topographies. In Society for Neuroscience Abstracts, 27:642.
Moore, J. W., Schmajuk, N. A. (2008). Kamin blocking. Scholarpedia, 3(5):3542.
Moore, J. W., Stickney, K. J. (1980). Formation of attentional-associative networks in realtime:Role of the hippocampus and implications for conditioning. Physiological Psychology,
8(2):207–217.
Mukundan, J., Martı́nez, J. F. (2012). MORSE, Multi-objective reconfigurable self-optimizing
memory scheduler. In IEEE 18th International Symposium on High Performance Computer
Architecture, pp. 1–12.
Müller, M. (2002). Computer Go. Artificial Intelligence, 134(1):145–179.
Munos, R., Stepleton, T., Harutyunyan, A., Bellemare, M. (2016). Safe and efficient o↵-policy
reinforcement learning. In Advances in Neural Information Processing Systems 29, pp. 1046–
1054. Curran Associates, Inc.
Naddaf, Y. (2010). Game-Independent AI Agents for Playing Atari 2600 Console Games. PhD
thesis, University of Alberta, Edmonton.
Narendra, K. S., Thathachar, M. A. L. (1974). Learning automata—A survey. IEEE Transactions
on Systems, Man, and Cybernetics, 4:323–334.
Narendra, K. S., Thathachar, M. A. L. (1989). Learning Automata: An Introduction. PrenticeHall, Englewood Cli↵s, NJ.

References

503

Narendra, K. S., Wheeler, R. M. (1983). An N-player sequential stochastic game with identicalpayo↵s. IEEE Transactions on Systems, Man, and Cybernetics, 6:1154–1158.
Narendra, K. S., Wheeler, R. M. (1986). Decentralized learning in finite Markov chains. IEEE
Transactions on Automatic Control, 31(6):519–526.
Nedić, A., Bertsekas, D. P. (2003). Least squares policy evaluation algorithms with linear
function approximation. Discrete Event Dynamic Systems, 13 (1-2):79–110.
Ng, A. Y. (2003). Shaping and Policy Search in Reinforcement Learning. PhD thesis, Universityof California, Berkeley.
Ng, A. Y., Harada, D., Russell, S. (1999). Policy invariance under reward transformations:
Theory and application to reward shaping. In I. Bratko and S. Dzeroski (Eds.), Proceedingsof the 16th International Conference on Machine Learning, pp. 278–287.
Ng, A. Y., Russell, S. J. (2000). Algorithms for inverse reinforcement learning. In Proceedings ofthe 17th International Conference on Machine Learning, pp. 663–670.
Niv, Y. (2009). Reinforcement learning in the brain. Journal of Mathematical Psychology,
53(3):139–154.
Niv, Y., Daw, N. D., Dayan, P. (2006). How fast to work: Response vigor, motivation and tonic
dopamine. In Advances in Neural Information Processing Systems 18, pp. 1019–1026. MIT
Press, Cambridge, MA.
Niv, Y., Daw, N. D., Joel, D., Dayan, P. (2007). Tonic dopamine: opportunity costs and thecontrol of response vigor. Psychopharmacology, 191(3):507–520.
Niv, Y., Joel, D., Dayan, P. (2006). A normative perspective on motivation. Trends in Cognitive
Sciences, 10(8):375–381.
Nouri, A., Littman, M. L. (2009). Multi-resolution exploration in continuous spaces. In Advancesin Neural Information Processing Systems 21, pp. 1209–1216. Curran Associates, Inc.
Nowé, A., Vrancx, P., Hauwere, Y.-M. D. (2012). Game theory and multi-agent reinforcement
learning. In M. Wiering and M. van Otterlo (Eds.), Reinforcement Learning: State-of-the-Art,
pp. 441–467. Springer-Verlag Berlin Heidelberg.
Nutt, D. J., Lingford-Hughes, A., Erritzoe, D., Stokes, P. R. A. (2015). The dopamine theory of
addiction: 40 years of highs and lows. Nature Reviews Neuroscience, 16(5):305–312.
O’Doherty, J. P., Dayan, P., Friston, K., Critchley, H., Dolan, R. J. (2003). Temporal di↵erence
models and reward-related learning in the human brain. Neuron, 38(2):329–337.
O’Doherty, J. P., Dayan, P., Schultz, J., Deichmann, R., Friston, K., Dolan, R. J. (2004).
Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science,
304(5669):452–454.
Ólafsdóttir, H. F., Barry, C., Saleem, A. B., Hassabis, D., Spiers, H. J. (2015). Hippocampal
place cells construct reward related sequences through unexplored space. Elife, 4:e06063.
Oh, J., Guo, X., Lee, H., Lewis, R. L., Singh, S. (2015). Action-conditional video prediction
using deep networks in Atari games. In Advances in Neural Information Processing Systems
28, pp. 2845–2853. Curran Associates, Inc.
Olds, J., Milner, P. (1954). Positive reinforcement produced by electrical stimulation of the septal
area and other regions of rat brain. Journal of Comparative and Physiological Psychology,
47(6):419–427.
O’Reilly, R. C., Frank, M. J. (2006). Making working memory work: A computational model oflearning in the prefrontal cortex and basal ganglia. Neural Computation, 18(2):283–328.
O’Reilly, R. C., Frank, M. J., Hazy, T. E., Watz, B. (2007). PVLV, the primary value andlearned value Pavlovian learning algorithm. Behavioral Neuroscience, 121(1):31–49.

504

References

Omohundro, S. M. (1987). Efficient algorithms with neural network behavior. Technical Report,
Department of Computer Science, University of Illinois at Urbana-Champaign.
Ormoneit, D., Sen, Ś. (2002). Kernel-based reinforcement learning. Machine Learning, 49 (23):161–178.
Oudeyer, P.-Y., Kaplan, F. (2007). What is intrinsic motivation? A typology of computational
approaches. Frontiers in Neurorobotics, 1:6.
Oudeyer, P.-Y., Kaplan, F., Hafner, V. V. (2007). Intrinsic motivation systems for autonomous
mental development. IEEE Transactions on Evolutionary Computation, 11(2):265–286.
Padoa-Schioppa, C., Assad, J. A. (2006). Neurons in the orbitofrontal cortex encode economic
value. Nature, 441 (7090):223–226.
Page, C. V. (1977). Heuristics for signature table analysis as a pattern recognition technique.
IEEE Transactions on Systems, Man, and Cybernetics, 7(2):77–86.
Pagnoni, G., Zink, C. F., Montague, P. R., Berns, G. S. (2002). Activity in human ventral
striatum locked to errors of reward prediction. Nature Neuroscience, 5(2):97–98.
Pan, W.-X., Schmidt, R., Wickens, J. R., Hyland, B. I. (2005). Dopamine cells respond to
predicted events during classical conditioning: Evidence for eligibility traces in the rewardlearning network. The Journal of Neuroscience, 25(26):6235–6242.
Park, J., Kim, J., Kang, D. (2005). An RLS-based natural actor–critic algorithm for locomotionof a two-linked robot arm. Computational Intelligence and Security:65–72.
Parks, P. C., Militzer, J. (1991). Improved allocation of weights for associative memory storage
in learning control systems. In IFAC Design Methods of Control Systems, Zurich, Switzerland,
pp. 507–512.
Parr, R. (1988). Hierarchical Control and Learning for Markov Decision Processes. PhD thesis,
University of California, Berkeley.
Parr, R., Li, L., Taylor, G., Painter-Wakefield, C., Littman, M. L. (2008). An analysis of linear
models, linear value-function approximation, and feature selection for reinforcement learning.
In Proceedings of the 25th international conference on Machine learning, pp. 752–759).
Parr, R., Russell, S. (1995). Approximating optimal policies for partially observable stochastic
domains. In Proceedings of the Fourteenth International Joint Conference on Artificial
Intelligence, pp. 1088–1094. Morgan Kaufmann.
Pavlov, I. P. (1927). Conditioned Reflexes. Oxford University Press, London.
Pawlak, V., Kerr, J. N. D. (2008). Dopamine receptor activation is required for corticostriatal
spike-timing-dependent plasticity. The Journal of Neuroscience, 28(10):2435–2446.
Pawlak, V., Wickens, J. R., Kirkwood, A., Kerr, J. N. D. (2010). Timing is not everything: neuromodulation opens the STDP gate. Frontiers in Synaptic Neuroscience, 2:146.
doi:10.3389/fnsyn.2010.00146.
Pearce, J. M., Hall, G. (1980). A model for Pavlovian learning: Variation in the e↵ectiveness ofconditioning but not unconditioned stimuli. Psychological Review, 87(6):532–552.
Pearl, J. (1984). Heuristics: Intelligent Search Strategies for Computer Problem Solving.
Addison-Wesley, Reading, MA.
Pearl, J. (1995). Causal diagrams for empirical research. Biometrika, 82(4):669-688.
Pecevski, D., Maass, W., Legenstein, R. A. (2008). Theoretical analysis of learning withreward-modulated spike-timing-dependent plasticity. In Advances in Neural Information
Processing Systems 20, pp. 881–888. Curran Associates, Inc.
Peng, J. (1993). Efficient Dynamic Programming-Based Learning for Control. PhD thesis,
Northeastern University, Boston MA.

References

505

Peng, J. (1995). Efficient memory-based dynamic programming. In Proceedings of the 12th
International Conference on Machine Learning, pp. 438–446.
Peng, J., Williams, R. J. (1993). Efficient learning and planning within the Dyna framework.
Adaptive Behavior, 1(4):437–454.
Peng, J., Williams, R. J. (1994). Incremental multi-step Q-learning. In Proceedings of the
11th International Conference on Machine Learning, pp. 226–232. Morgan Kaufmann, San
Francisco.
Peng, J., Williams, R. J. (1996). Incremental multi-step Q-learning. Machine Learning,22(1):283–290.
Perkins, T. J., Pendrith, M. D. (2002). On the existence of fixed points for Q-learning andSarsa in partially observable domains. In Proceedings of the 19th International Conference
on Machine Learning, pp. 490–497.
Perkins, T. J., Precup, D. (2003). A convergent form of approximate policy iteration. In Advancesin Neural Information Processing Systems 15, pp. 1627–1634. MIT Press, Cambridge, MA.
Peters, J., Büchel, C. (2010). Neural representations of subjective reward value. Behavioral
Brain Research, 213(2):135–141.
Peters, J., Schaal, S. (2008). Natural actor–critic. Neurocomputing, 71 (7):1180–1190.
Peters, J., Vijayakumar, S., Schaal, S. (2005). Natural actor–critic. In European Conference on
Machine Learning, pp. 280–291. Springer Berlin Heidelberg.
Pezzulo, G., van der Meer, M. A. A., Lansink, C. S., Pennartz, C. M. A. (2014). Internallygenerated sequences in learning and executing goal-directed behavior. Trends in Cognitive
Science, 18(12):647–657.
Pfei↵er, B. E., Foster, D. J. (2013). Hippocampal place-cell sequences depict future paths toremembered goals. Nature, 497(7447):74–79.
Phansalkar, V. V., Thathachar, M. A. L. (1995). Local and global optimization algorithms forgeneralized learning automata. Neural Computation, 7(5):950–973.
Poggio, T., Girosi, F. (1989). A theory of networks for approximation and learning. A.I. Memo1140. Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge,
MA.
Poggio, T., Girosi, F. (1990). Regularization algorithms for learning that are equivalent tomultilayer networks. Science, 247(4945):978–982.
Polyak, B. T. (1990). New stochastic approximation type procedures. Automat. i Telemekh,
7 (98-107):2 (in Russian).
Polyak, B. T., Juditsky, A. B. (1992). Acceleration of stochastic approximation by averaging.
SIAM Journal on Control and Optimization, 30 (4):838–855.
Powell, M. J. D. (1987). Radial basis functions for multivariate interpolation: A review. In
J. C. Mason and M. G. Cox (Eds.), Algorithms for Approximation, pp. 143–167. Clarendon
Press, Oxford.
Powell, W. B. (2011). Approximate Dynamic Programming: Solving the Curses of Dimensionality,
Second edition. John Wiley and Sons.
Powers, W. T. (1973). Behavior: The Control of Perception. Aldine de Gruyter, Chicago. 2nd
expanded edition 2005.
Precup, D. (2000). Temporal Abstraction in Reinforcement Learning. PhD thesis, University of
Massachusetts, Amherst.
Precup, D., Sutton, R. S., Dasgupta, S. (2001). O↵-policy temporal-di↵erence learning withfunction approximation. In Proceedings of the 18th International Conference on Machine
Learning, pp. 417–424.

506

References

Precup, D., Sutton, R. S., Paduraru, C., Koop, A., Singh, S. (2006). O↵-policy learningwith options and recognizers. In Advances in Neural Information Processing Systems 18,
pp. 1097–1104. MIT Press, Cambridge, MA.
Precup, D., Sutton, R. S., Singh, S. (2000). Eligibility traces for o↵-policy policy evaluation. In
Proceedings of the 17th International Conference on Machine Learning, pp. 759–766. Morgan
Kaufmann.
Puterman, M. L. (1994). Markov Decision Problems. Wiley, New York.
Puterman, M. L., Shin, M. C. (1978). Modified policy iteration algorithms for discounted
Markov decision problems. Management Science, 24(11):1127–1137.
Quartz, S., Dayan, P., Montague, P. R., Sejnowski, T. J. (1992). Expectation learning in thebrain using di↵use ascending connections. In Society for Neuroscience Abstracts, 18:1210.
Randløv, J., Alstrøm, P. (1998). Learning to drive a bicycle using reinforcement learningand shaping. In Proceedings of the 15th International Conference on Machine Learning,pp. 463–471.
Rangel, A., Camerer, C., Montague, P. R. (2008). A framework for studying the neurobiology ofvalue-based decision making. Nature Reviews Neuroscience, 9(7):545–556.
Rangel, A., Hare, T. (2010). Neural computations associated with goal-directed choice. Current
Opinion in Neurobiology, 20(2):262–270.
Rao, R. P., Sejnowski, T. J. (2001). Spike-timing-dependent Hebbian plasticity as temporaldi↵erence learning. Neural Computation, 13 (10):2221–2237.
Ratcli↵, R. (1990). Connectionist models of recognition memory: Constraints imposed bylearning and forgetting functions. Psychological Review, 97 (2):285–308.
Reddy, G., Celani, A., Sejnowski, T. J., Vergassola, M. (2016). Learning to soar in turbulent
environments. Proceedings of the National Academy of Sciences, 113(33):E4877–E4884.
Redish, D. A. (2004). Addiction as a computational process gone awry. Science, 306(5703):1944–
1947.
Reetz, D. (1977). Approximate solutions of a discounted Markovian decision process. BonnerMathematische Schriften, 98:77–92.
Rescorla, R. A., Wagner, A. R. (1972). A theory of Pavlovian conditioning: Variations in thee↵ectiveness of reinforcement and nonreinforcement. In A. H. Black and W. F. Prokasy(Eds.), Classical Conditioning II, pp. 64–99. Appleton-Century-Crofts, New York.
Revusky, S., Garcia, J. (1970). Learned associations over long delays. In G. Bower (Ed.), The
Psychology of Learning and Motivation, v. 4, pp. 1–84. Academic Press, Inc., New York.
Reynolds, J. N. J., Wickens, J. R. (2002). Dopamine-dependent plasticity of corticostriatal
synapses. Neural Networks, 15(4):507–521.
Ring, M. B. (in preparation). Representing knowledge as forecasts (and state as knowledge).
Ripley, B. D. (2007). Pattern Recognition and Neural Networks. Cambridge University Press.
Rixner, S. (2004). Memory controller optimizations for web servers. In Proceedings of the37th annual IEEE/ACM International Symposium on Microarchitecture, p. 355–366. IEEE
Computer Society.
Robbins, H. (1952). Some aspects of the sequential design of experiments. Bulletin of theAmerican Mathematical Society, 58:527–535.
Robertie, B. (1992). Carbon versus silicon: Matching wits with TD-Gammon. Inside Backgammon, 2(2):14–22.
Romo, R., Schultz, W. (1990). Dopamine neurons of the monkey midbrain: Contingencies of
responses to active touch during self-initiated arm movements. Journal of Neurophysiology,
63(3):592–624.

References

507

Rosenblatt, F. (1962). Principles of Neurodynamics: Perceptrons and the Theory of Brain
Mechanisms. Spartan Books, Washington, DC.
Ross, S. (1983). Introduction to Stochastic Dynamic Programming. Academic Press, New York.
Ross, T. (1933). Machines that think. Scientific American, 148(4):206–208.
Rubinstein, R. Y. (1981). Simulation and the Monte Carlo Method. Wiley, New York.
Rumelhart, D. E., Hinton, G. E., Williams, R. J. (1986). Learning internal representations byerror propagation. In D. E. Rumelhart and J. L. McClelland (Eds.), Parallel Distributed Processing: Explorations in the Microstructure of Cognition, vol. I, Foundations. Bradford/MIT
Press, Cambridge, MA.
Rummery, G. A. (1995). Problem Solving with Reinforcement Learning. PhD thesis, Universityof Cambridge.
Rummery, G. A., Niranjan, M. (1994). On-line Q-learning using connectionist systems. Technical
Report CUED/F-INFENG/TR 166. Engineering Department, Cambridge University.
Ruppert, D. (1988). Efficient estimations from a slowly convergent Robbins-Monro process.
Cornell University Operations Research and Industrial Engineering Technical Report No. 781.
Russell, S., Norvig, P. (2009). Artificial Intelligence: A Modern Approach, 3rd edition. PrenticeHall, Englewood Cli↵s, NJ.
Russo, D. J., Van Roy, B., Kazerouni, A., Osband, I., Wen, Z. (2018). A tutorial on Thompson
sampling, Foundations and Trends in Machine Learning. ArXiv:1707.02038.
Rust, J. (1996). Numerical dynamic programming in economics. In H. Amman, D. Kendrick, andJ. Rust (Eds.), Handbook of Computational Economics, pp. 614–722. Elsevier, Amsterdam.
Saddoris, M. P., Cacciapaglia, F., Wightmman, R. M., Carelli, R. M. (2015). Di↵erential
dopamine release dynamics in the nucleus accumbens core and shell reveal complementary signals for error prediction and incentive motivation. The Journal of Neuroscience,
35(33):11572–11582.
Saksida, L. M., Raymond, S. M., Touretzky, D. S. (1997). Shaping robot behavior using principles
from instrumental conditioning. Robotics and Autonomous Systems, 22(3):231–249.
Samuel, A. L. (1959). Some studies in machine learning using the game of checkers. IBM
Journal on Research and Development, 3(3), 210–229.
Samuel, A. L. (1967). Some studies in machine learning using the game of checkers. II—Recent
progress. IBM Journal on Research and Development, 11(6):601–617.
Schaal, S., Atkeson, C. G. (1994). Robot juggling: Implementation of memory-based learning.
IEEE Control Systems, 14 (1):57–71.
Schmajuk, N. A. (2008). Computational models of classical conditioning. Scholarpedia,
3(3):1664.
Schmidhuber, J. (1991a). Curious model-building control systems. In Proceedings of the IEEE
International Joint Conference on Neural Networks, pp. 1458–1463. IEEE.
Schmidhuber, J. (1991b). A possibility for implementing curiosity and boredom in model-building
neural controllers. In From Animals to Animats: Proceedings of the First International
Conference on Simulation of Adaptive Behavior, pp. 222–227. MIT Press, Cambridge, MA.
Schmidhuber, J. (2015). Deep learning in neural networks: An overview. Neural Networks,
6 :85–117.
Schmidhuber, J., Storck, J., Hochreiter, S. (1994). Reinforcement driven information acquisition
in nondeterministic environments. Technical report, Fakultät für Informatik, Technische
Universität München, München, Germany.
Schraudolph, N. N. (1999). Local gain adaptation in stochastic gradient descent. In Proceedingsof the International Conference on Artificial Neural Networks, pp. 569–574. IEEE, London.

508

References

Schraudolph, N. N. (2002). Fast curvature matrix-vector products for second-order gradientdescent. Neural Computation, 14 (7):1723–1738.
Schraudolph, N. N., Yu, J., Aberdeen, D. (2006). Fast online policy gradient learning with SMD
gain vector adaptation. In Advances in Neural Information Processing Systems, pp. 1185–
1192.
Schulman, J., Chen, X., Abbeel, P. (2017). Equivalence between policy gradients and soft
Q-Learning. ArXiv:1704.06440.
Schultz, D. G., Melsa, J. L. (1967). State Functions and Linear Control Systems. McGraw-Hill,
New York.
Schultz, W. (1998). Predictive reward signal of dopamine neurons. Journal of Neurophysiology,
80(1):1–27.
Schultz, W., Apicella, P., Ljungberg, T. (1993). Responses of monkey dopamine neurons toreward and conditioned stimuli during successive steps of learning a delayed response task.
The Journal of Neuroscience, 13 (3):900–913.
Schultz, W., Dayan, P., Montague, P. R. (1997). A neural substrate of prediction and reward.
Science, 275(5306):1593–1598.
Schultz, W., Romo, R. (1990). Dopamine neurons of the monkey midbrain: contingencies of
responses to stimuli eliciting immediate behavioral reactions. Journal of Neurophysiology,
63(3):607–624.
Schultz, W., Romo, R., Ljungberg, T., Mirenowicz, J., Hollerman, J. R., Dickinson, A. (1995).
Reward-related signals carried by dopamine neurons. In J. C. Houk, J. L. Davis, and D. G.
Beiser (Eds.), Models of Information Processing in the Basal Ganglia, pp. 233–248. MIT
Press, Cambridge, MA.
Schwartz, A. (1993). A reinforcement learning method for maximizing undiscounted rewards. In
Proceedings of the 10th International Conference on Machine Learning, pp. 298–305. Morgan
Kaufmann.
Schweitzer, P. J., Seidmann, A. (1985). Generalized polynomial approximations in Markovian
decision processes. Journal of Mathematical Analysis and Applications, 110(2):568–582.
Selfridge, O. G. (1978). Tracking and trailing: Adaptation in movement strategies. Technicalreport, Bolt Beranek and Newman, Inc. Unpublished report.
Selfridge, O. G. (1984). Some themes and primitives in ill-defined systems. In O. G. Selfridge,
E. L. Rissland, and M. A. Arbib (Eds.), Adaptive Control of Ill-Defined Systems, pp. 21–26.
Plenum Press, NY. Proceedings of the NATO Advanced Research Institute on Adaptive
Control of Ill-defined Systems, NATO Conference Series II, Systems Science, Vol. 16.
Selfridge, O. J., Sutton, R. S., Barto, A. G. (1985). Training and tracking in robotics. In A. Joshi(Ed.), Proceedings of the Ninth International Joint Conference on Artificial Intelligence,
pp. 670–672. Morgan Kaufmann.
Seo, H., Barraclough, D., Lee, D. (2007). Dynamic signals related to choices and outcomes inthe dorsolateral prefrontal cortex. Cerebral Cortex, 17(suppl 1):110–117.
Seung, H. S. (2003). Learning in spiking neural networks by reinforcement of stochastic synaptictransmission. Neuron, 40(6):1063–1073.
Shah, A. (2012). Psychological and neuroscientific connections with reinforcement learning. In M.
Wiering and M. van Otterlo (Eds.), Reinforcement Learning: State-of-the-Art, pp. 507–537.
Springer-Verlag Berlin Heidelberg.
Shannon, C. E. (1950). Programming a computer for playing chess. Philosophical Magazine andJournal of Science, 41(314):256–275.

References

509

Shannon, C. E. (1951). Presentation of a maze-solving machine. In H. V. Forester (Ed.), Cybernetics. Transactions of the Eighth Conference, pp. 173–180. Josiah Macy Jr. Foundation.
Shannon, C. E. (1952). “Theseus” maze-solving mouse. http://cyberneticzoo.com/mazesolvers/1952—
theseus-maze-solving-mouse—claude-shannon-american/.
Shelton, C. R. (2001). Importance Sampling for Reinforcement Learning with Multiple Objectives.
PhD thesis, Massachusetts Institute of Technology, Cambridge MA.
Shepard, D. (1968). A two-dimensional interpolation function for irregularly-spaced data. In
Proceedings of the 23rd ACM National Conference, pp. 517–524. ACM, New York.
Sherman, J., Morrison, W. J. (1949). Adjustment of an inverse matrix corresponding to changes
in the elements of a given column or a given row of the original matrix (abstract). Annals of
Mathematical Statistics, 20 (4):621.
Shewchuk, J., Dean, T. (1990). Towards learning time-varying functions with high input
dimensionality. In Proceedings of the Fifth IEEE International Symposium on Intelligent
Control, pp. 383–388. IEEE Computer Society Press, Los Alamitos, CA.
Shimansky, Y. P. (2009). Biologically plausible learning in neural networks: a lesson frombacterial chemotaxis. Biological Cybernetics, 101(5-6):379–385.
Si, J., Barto, A., Powell, W., Wunsch, D. (Eds.) (2004). Handbook of Learning and Approximate
Dynamic Programming. John Wiley and Sons.
Silver, D. (2009). Reinforcement Learning and Simulation Based Search in the Game of Go.
PhD thesis, University of Alberta, Edmonton.
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser,J., Antonoglou, I., Panneershelvam, V., Lanctot, M., Dieleman, S., Grewe, D., Nham, J.,
Kalchbrenner, N., Sutskever, I., Lillicrap, T., Leach, M., Kavukcuoglu, K., Graepel, T.,
Hassabis, D. (2016). Mastering the game of Go with deep neural networks and tree search.
Nature, 529(7587):484–489.
Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., Riedmiller, M. (2014). Deterministic
policy gradient algorithms. In Proceedings of the 31st International Conference on Machine
Learning, pp. 387–395.
Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T.,
Baker, L., Lai, M., Bolton, A., Chen, Y., Lillicrap, L., Hui, F., Sifre, L., van den Driessche,G., Graepel, T., Hassibis, D. (2017a). Mastering the game of Go without human knowledge.
Nature, 550 (7676):354–359.
Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L.,
Kumaran, D., Graepel, T., Lillicrap, T., Simoyan, K., Hassibis, D. (2017b). Mastering chessand shogi by self-play with a general reinforcement learning algorithm. ArXiv:1712.01815.
Şimşek, Ö., Algórta, S., Kothiyal, A. (2016). Why most decisions are easy in tetris—And perhaps
in other sequential decision problems, as well. In Proceedings of the 33rd International
Conference on Machine Learning, pp. 1757-1765.
Simon, H. (2000). Lecture at the Earthware Symposium, Carnegie Mellon University. https://www.youtube.com/watch?v=EZhyi-8DBjc.
Singh, S. P. (1992a). Reinforcement learning with a hierarchy of abstract models. In Proceedingsof the Tenth National Conference on Artificial Intelligence, pp. 202–207. AAAI/MIT Press,
Menlo Park, CA.
Singh, S. P. (1992b). Scaling reinforcement learning algorithms by learning variable temporal
resolution models. In Proceedings of the 9th International Workshop on Machine Learning,pp. 406–415. Morgan Kaufmann.
Singh, S. P. (1993). Learning to Solve Markovian Decision Processes. PhD thesis, University of
Massachusetts, Amherst.

510

References

Singh, S. P. (Ed.) (2002). Special double issue on reinforcement learning, Machine Learning,49(2-3).
Singh, S., Barto, A. G., Chentanez, N. (2005). Intrinsically motivated reinforcement learning.
In Advances in Neural Information Processing Systems 17, pp. 1281–1288. MIT Press,
Cambridge, MA.
Singh, S. P., Bertsekas, D. (1997). Reinforcement learning for dynamic channel allocationin cellular telephone systems. In Advances in Neural Information Processing Systems 9,
pp. 974–980. MIT Press, Cambridge, MA.
Singh, S. P., Jaakkola, T., Jordan, M. I. (1994). Learning without state-estimation in partiallyobservable Markovian decision problems. In Proceedings of the 11th International Conference
on Machine Learning, pp. 284–292. Morgan Kaufmann.
Singh, S., Jaakkola, T., Littman, M. L., Szepesvári, C. (2000). Convergence results for single-step
on-policy reinforcement-learning algorithms. Machine Learning, 38 (3):287–308.
Singh, S. P., Jaakkola, T., Jordan, M. I. (1995). Reinforcement learning with soft state
aggregation. In Advances in Neural Information Processing Systems 7, pp. 359–368. MIT
Press, Cambridge, MA.
Singh, S., Lewis, R. L., Barto, A. G. (2009). Where do rewards come from? In N. Taatgenand H. van Rijn (Eds.), Proceedings of the 31st Annual Conference of the Cognitive Science
Society, pp. 2601–2606. Cognitive Science Society.
Singh, S., Lewis, R. L., Barto, A. G., Sorg, J. (2010). Intrinsically motivated reinforcement
learning: An evolutionary perspective. IEEE Transactions on Autonomous Mental Development, 2(2):70–82. Special issue on Active Learning and Intrinsically Motivated Explorationin Robots: Advances and Challenges.
Singh, S. P., Sutton, R. S. (1996). Reinforcement learning with replacing eligibility traces.
Machine Learning, 22(1-3):123–158.
Skinner, B. F. (1938). The Behavior of Organisms: An Experimental Analysis. Appleton-Century,
New York.
Skinner, B. F. (1958). Reinforcement today. American Psychologist, 13(3):94–99.
Skinner, B. F. (1963). Operant behavior. American Psychologist, 18 (8):503–515.
Sofge, D. A., White, D. A. (1992). Applied learning: Optimal control for manufacturing. In
D. A. White and D. A. Sofge (Eds.), Handbook of Intelligent Control: Neural, Fuzzy, andAdaptive Approaches, pp. 259–281. Van Nostrand Reinhold, New York.
Sorg, J. D. (2011). The Optimal Reward Problem:Designing E↵ective Reward for Bounded
Agents. PhD thesis, University of Michigan, Ann Arbor.
Sorg, J., Lewis, R. L., Singh, S. P. (2010). Reward design via online gradient ascent. In Advancesin Neural Information Processing Systems 23, pp. 2190–2198. Curran Associates, Inc.
Sorg, J., Singh, S. (2010). Linear options. In Proceedings of the 9th International Conference on
Autonomous Agents and Multiagent Systems, pp. 31–38.
Sorg, J., Singh, S., Lewis, R. (2010). Internal rewards mitigate agent boundedness. In Proceedingsof the 27th International Conference on Machine Learning, pp. 1007–1014.
Spence, K. W. (1947). The role of secondary reinforcement in delayed reward learning. Psychological Review, 54(1):1–8.
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R. (2014). Dropout:
A simple way to prevent neural networks from overfitting. Journal of Machine Learning
Research, 15(1):1929–1958.
Staddon, J. E. R. (1983). Adaptive Behavior and Learning. Cambridge University Press.

References

511

Stanfill, C., Waltz, D. (1986). Toward memory-based reasoning. Communications of the ACM,
29 (12):1213–1228.
Steinberg, E. E., Keiflin, R., Boivin, J. R., Witten, I. B., Deisseroth, K., Janak, P. H. (2013). A
causal link between prediction errors, dopamine neurons and learning. Nature Neuroscience,16(7):966–973.
Sterling, P., Laughlin, S. (2015). Principles of Neural Design. MIT Press, Cambridge, MA.
Sternberg, S. (1963). Stochastic learning theory. In: Handbook of Mathematical Psychology,
Volume II, R. D. Luce, R. R. Bush, and E. Galanter (Eds.). John Wiley & Sons.
Sugiyama, M., Hachiya, H., Morimura, T. (2013). Statistical Reinforcement Learning: Modern
Machine Learning Approaches. Chapman & Hall/CRC.
Suri, R. E., Bargas, J., Arbib, M. A. (2001). Modeling functions of striatal dopamine modulationin learning and planning. Neuroscience, 103(1):65–85.
Suri, R. E., Schultz, W. (1998). Learning of sequential movements by neural network modelwith dopamine-like reinforcement signal. Experimental Brain Research, 121(3):350–354.
Suri, R. E., Schultz, W. (1999). A neural network model with dopamine-like reinforcement
signal that learns a spatial delayed response task. Neuroscience, 91(3):871–890.
Sutton, R. S. (1978a). Learning theory support for a single channel theory of the brain.
Unpublished report.
Sutton, R. S. (1978b). Single channel theory: A neuronal theory of learning. Brain Theory
Newsletter, 4:72–75. Center for Systems Neuroscience, University of Massachusetts, Amherst,MA.
Sutton, R. S. (1978c). A unified theory of expectation in classical and instrumental conditioning.
Bachelors thesis, Stanford University.
Sutton, R. S. (1984). Temporal Credit Assignment in Reinforcement Learning. PhD thesis,
University of Massachusetts, Amherst.
Sutton, R. S. (1988). Learning to predict by the method of temporal di↵erences. Machine
Learning, 3(1):9–44 (important erratum p. 377).
Sutton, R. S. (1990). Integrated architectures for learning, planning, and reacting based on
approximating dynamic programming. In Proceedings of the 7th International Workshop on
Machine Learning, pp. 216–224. Morgan Kaufmann.
Sutton, R. S. (1991a). Dyna, an integrated architecture for learning, planning, and reacting.
SIGART Bulletin, 2(4):160–163. ACM, New York.
Sutton, R. S. (1991b). Planning by incremental dynamic programming. In Proceedings of the
8th International Workshop on Machine Learning, pp. 353–357. Morgan Kaufmann.
Sutton, R. S. (Ed.) (1992a). Reinforcement Learning. Kluwer Academic Press. Reprinting of a
special double issue on reinforcement learning, Machine Learning, 8(3-4).
Sutton, R. S. (1992b). Adapting bias by gradient descent: An incremental version of delta-bardelta. Proceedings of the Tenth National Conference on Artificial Intelligence, pp. 171–176,
MIT Press.
Sutton, R. S. (1992c). Gain adaptation beats least squares? Proceedings of the Seventh Yale
Workshop on Adaptive and Learning Systems, pp. 161–166, Yale University, New Haven, CT.
Sutton, R. S. (1995a). TD models: Modeling the world at a mixture of time scales. In
Proceedings of the 12th International Conference on Machine Learning, pp. 531–539. Morgan
Kaufmann.
Sutton, R. S. (1995b). On the virtues of linear learning and trajectory distributions. In
Proceedings of the Workshop on Value Function Approximation at The 12th International
Conference on Machine Learning.

512

References

Sutton, R. S. (1996). Generalization in reinforcement learning: Successful examples using sparsecoarse coding. In Advances in Neural Information Processing Systems 8, pp. 1038–1044.
MIT Press, Cambridge, MA.
Sutton, R. S. (2009). The grand challenge of predictive empirical abstract knowledge. Working
Notes of the IJCAI-09 Workshop on Grand Challenges for Reasoning from Experiences.
Sutton, R. S. (2015a) Introduction to reinforcement learning with function approximation.
Tutorial at the Conference on Neural Information Processing Systems, Montreal, December
7, 2015.
Sutton, R. S. (2015b) True online Emphatic TD( ): Quick reference and implementation guide.
ArXiv:1507.07147. Code is available in Python and C++ by downloading the source files ofthis arXiv paper as a zip archive.
Sutton, R. S., Barto, A. G. (1981a). Toward a modern theory of adaptive networks: Expectationand prediction. Psychological Review, 88(2):135–170.
Sutton, R. S., Barto, A. G. (1981b). An adaptive network that constructs and uses an internal
model of its world. Cognition and Brain Theory, 3:217–246.
Sutton, R. S., Barto, A. G. (1987). A temporal-di↵erence model of classical conditioning. In
Proceedings of the Ninth Annual Conference of the Cognitive Science Society, pp. 355-378.
Erlbaum, Hillsdale, NJ.
Sutton, R. S., Barto, A. G. (1990). Time-derivative models of Pavlovian reinforcement. In
M. Gabriel and J. Moore (Eds.), Learning and Computational Neuroscience: Foundations of
Adaptive Networks, pp. 497–537. MIT Press, Cambridge, MA.
Sutton, R. S., Maei, H. R., Precup, D., Bhatnagar, S., Silver, D., Szepesvári, Cs., Wiewiora, E.
(2009a). Fast gradient-descent methods for temporal-di↵erence learning with linear function
approximation. In Proceedings of the 26th International Conference on Machine Learning,
pp. 993–1000. ACM, New York.
Sutton, R. S., Szepesvári, Cs., Maei, H. R. (2009b). A convergent O(d2 ) temporal-di↵erence
algorithm for o↵-policy learning with linear function approximation. In Advances in Neural
Information Processing Systems 21, pp. 1609–1616. Curran Associates, Inc.
Sutton, R. S., Mahmood, A. R., Precup, D., van Hasselt, H. (2014). A new Q( ) with interim
forward view and Monte Carlo equivalence. In Proceedings of the International Conference
on Machine Learning, 31. JMLR W&CP 32 (2).
Sutton, R. S., Mahmood, A. R., White, M. (2016). An emphatic approach to the problem ofo↵-policy temporal-di↵erence learning. Journal of Machine Learning Research, 17 (73):1–29.
Sutton, R. S., McAllester, D. A., Singh, S. P., Mansour, Y. (2000). Policy gradient methods
for reinforcement learning with function approximation. In Advances in Neural Information
Processing Systems 12, pp. 1057–1063. MIT Press, Cambridge, MA.
Sutton, R. S., Modayil, J., Delp, M., Degris, T., Pilarski, P. M., White, A., Precup, D. (2011).
Horde: A scalable real-time architecture for learning knowledge from unsupervised sensorimotor interaction. In Proceedings of the Tenth International Conference on Autonomous
Agents and Multiagent Systems, pp. 761–768, Taipei, Taiwan.
Sutton, R. S., Pinette, B. (1985). The learning of world models by connectionist networks. In
Proceedings of the Seventh Annual Conference of the Cognitive Science Society, pp. 54–64.
Sutton, R. S., Precup, D., Singh, S. (1999). Between MDPs and semi-MDPs: A framework for
temporal abstraction in reinforcement learning. Artificial Intelligence, 112 (1-2):181–211.
Sutton, R. S., Rafols, E., Koop, A. (2006). Temporal abstraction in temporal-di↵erence networks.
In Advances in neural information processing systems, pp. 1313–1320.
Sutton, R. S., Singh, S. P., McAllester, D. A. (2000). Comparing policy-gradient algorithms.
Unpublished manuscript.

References

513

Sutton, R. S., Szepesvári, Cs., Geramifard, A., Bowling, M., (2008). Dyna-style planning withlinear function approximation and prioritized sweeping. In Proceedings of the 24th Conference
on Uncertainty in Artificial Intelligence, pp. 528–536.
Sutton, R. S., Tanner, B. (2005). Temporal-di↵erence networks. In Advances in Neural
Information Processing Systems 17, p. 1377–1384.
Szepesvári, Cs. (2010). Algorithms for reinforcement learning. In Synthesis Lectures on Artificial
Intelligence and Machine Learning, 4(1):1–103. Morgan and Claypool.
Szita, I. (2012). Reinforcement learning in games. In M. Wiering and M. van Otterlo (Eds.),
Reinforcement Learning: State-of-the-Art, pp. 539–577. Springer-Verlag Berlin Heidelberg.
Tadepalli, P., Ok, D. (1994). H-learning: A reinforcement learning method to optimize
undiscounted average reward. Technical Report 94-30-01. Oregon State University, Computer
Science Department, Corvallis.
Tadepalli, P., Ok, D. (1996). Scaling up average reward reinforcement learning by approximating
the domain models and the value function. In Proceedings of the 13th International Conference
on Machine Learning, pp. 471–479.
Takahashi, Y., Schoenbaum, G., and Niv, Y. (2008). Silencing the critics: Understanding thee↵ects of cocaine sensitization on dorsolateral and ventral striatum in the context of anactor/critic model. Frontiers in Neuroscience, 2(1):86–99.
Tambe, M., Newell, A., Rosenbloom, P. S. (1990). The problem of expensive chunks and its
solution by restricting expressiveness. Machine Learning, 5 (3):299–348.
Tan, M. (1991). Learning a cost-sensitive internal representation for reinforcement learning. In
L. A. Birnbaum and G. C. Collins (Eds.), Proceedings of the 8th International Workshop on
Machine Learning, pp. 358–362. Morgan Kaufmann.
Tanner, B. (2006). Temporal-Di↵erence Networks. MSc thesis, University of Alberta.
Taylor, G., Parr, R. (2009). Kernelized value function approximation for reinforcement learning.
In Proceedings of the 26th International Conference on Machine Learning, pp. 1017–1024.
ACM, New York.
Taylor, M. E., Stone, P. (2009). Transfer learning for reinforcement learning domains: A survey.
Journal of Machine Learning Research, 10 :1633–1685.
Tesauro, G. (1986). Simple neural models of classical conditioning. Biological Cybernetics,
55(2-3):187–200.
Tesauro, G. (1992). Practical issues in temporal di↵erence learning. Machine Learning,8(3-4):257–277.
Tesauro, G. (1994). TD-Gammon, a self-teaching backgammon program, achieves master-levelplay. Neural Computation, 6(2):215–219.
Tesauro, G. (1995). Temporal di↵erence learning and TD-Gammon. Communications of theACM, 38(3):58–68.
Tesauro, G. (2002). Programming backgammon using self-teaching neural nets. Artificial
Intelligence, 134(1-2):181–199.
Tesauro, G., Galperin, G. R. (1997). On-line policy improvement using Monte-Carlo search. In
Advances in Neural Information Processing Systems 9, pp. 1068–1074. MIT Press, Cambridge,
MA.
Tesauro, G., Gondek, D. C., Lechner, J., Fan, J., Prager, J. M. (2012). Simulation, learning,and optimization techniques in Watson’s game strategies. IBM Journal of Research andDevelopment, 56(3-4):16–1–16–11.
Tesauro, G., Gondek, D. C., Lenchner, J., Fan, J., Prager, J. M. (2013). Analysis of Watson’s
strategies for playing Jeopardy! Journal of Artificial Intelligence Research, 47:205–251.
Tham, C. K. (1994). Modular On-Line Function Approximation for Scaling up Reinforcement
Learning. PhD thesis, University of Cambridge.

514

References

Thathachar, M. A. L., Sastry, P. S. (1985). A new approach to the design of reinforcement
schemes for learning automata. IEEE Transactions on Systems, Man, and Cybernetics,
15(1):168–175.
Thathachar, M., Sastry, P. S. (2002). Varieties of learning automata: an overview. IEEE
Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics, 36(6):711–722.
Thathachar, M., Sastry, P. S. (2011). Networks of Learning Automata: Techniques for Online
Stochastic Optimization. Springer Science & Business Media.
Theocharous, G., Thomas, P. S., Ghavamzadeh, M. (2015). Personalized ad recommendation forlife-time value optimization guarantees. In Proceedings of the Twenty-Fourth International
Joint Conference on Artificial Intelligence. AAAI Press, Palo Alto, CA.
Thistlethwaite, D. (1951). A critical review of latent learning and related experiments. Psychological Bulletin, 48(2):97–129.
Thomas, P. S. (2014). Bias in natural actor–critic algorithms. In Proceedings of the 31st
International Conference on Machine Learning, JMLR W&CP 32 (1), pp. 441–448.
Thomas, P. S. (2015). Safe Reinforcement Learning. PhD thesis, University of Massachusetts,
Amherst.
Thomas, P. S., Brunskill, E. (2017). Policy gradient methods for reinforcement learning withfunction approximation and action-dependent baselines. ArXiv:1706.06643.
Thomas, P. S., Theocharous, G., Ghavamzadeh, M. (2015). High-confidence o↵-policy evaluation.
In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, pp. 3000–3006.
AAAI Press, Menlo Park, CA.
Thompson, W. R. (1933). On the likelihood that one unknown probability exceeds another inview of the evidence of two samples. Biometrika, 25(3-4):285–294.
Thompson, W. R. (1934). On the theory of apportionment. American Journal of Mathematics,
57: 450–457.
Thon, M. (2017). Spectral Learning of Sequential Systems. PhD thesis, Jacobs University
Bremen.
Thon, M., Jaeger, H. (2015). Links between multiplicity automata, observable operator models
and predictive state representations: a unified learning framework. The Journal of Machine
Learning Research, 16 (1):103–147.
Thorndike, E. L. (1898). Animal intelligence: An experimental study of the associative processes
in animals. The Psychological Review, Series of Monograph Supplements, II(4).
Thorndike, E. L. (1911). Animal Intelligence. Hafner, Darien, CT.
Thorp, E. O. (1966). Beat the Dealer: A Winning Strategy for the Game of Twenty-One.
Random House, New York.
Tian, T. (in preparation) An Empirical Study of Sliding-Step Methods in Temporal Di↵erence
Learning. M.Sc thesis, University of Alberta, Edmonton.
Tieleman, T., Hinton, G. (2012). Lecture 6.5–RMSProp. COURSERA: Neural networks formachine learning 4.2:26–31.
Tolman, E. C. (1932). Purposive Behavior in Animals and Men. Century, New York.
Tolman, E. C. (1948). Cognitive maps in rats and men. Psychological Review, 55(4):189–208.
Tsai, H.-S., Zhang, F., Adamantidis, A., Stuber, G. D., Bonci, A., de Lecea, L., Deisseroth,K. (2009). Phasic firing in dopaminergic neurons is sufficient for behavioral conditioning.
Science, 324(5930):1080–1084.
Tsetlin, M. L. (1973). Automaton Theory and Modeling of Biological Systems. Academic Press,
New York.

References

515

Tsitsiklis, J. N. (1994). Asynchronous stochastic approximation and Q-learning. Machine
Learning, 16(3):185–202.
Tsitsiklis, J. N. (2002). On the convergence of optimistic policy iteration. Journal of Machine
Learning Research, 3:59–72.
Tsitsiklis, J. N., Van Roy, B. (1996). Feature-based methods for large scale dynamic programming.
Machine Learning, 22(1-3):59–94.
Tsitsiklis, J. N., Van Roy, B. (1997). An analysis of temporal-di↵erence learning with function
approximation. IEEE Transactions on Automatic Control, 42(5):674–690.
Tsitsiklis, J. N., Van Roy, B. (1999). Average cost temporal-di↵erence learning. Automatica,
35(11):1799–1808.
Turing, A. M. (1948). Intelligent machinery. In B. Jack Copeland (Ed.) (2004), The Essential
Turing, pp. 410–432. Oxford University Press, Oxford.
Ungar, L. H. (1990). A bioreactor benchmark for adaptive network-based process control. In
W. T. Miller, R. S. Sutton, and P. J. Werbos (Eds.), Neural Networks for Control, pp. 387–402.
MIT Press, Cambridge, MA.
Unnikrishnan, K. P., Venugopal, K. P. (1994). Alopex: A correlation-based learning algorithm
for feedforward and recurrent neural networks. N eural Computation, 6(3): 469–490.
Urbanczik, R., Senn, W. (2009). Reinforcement learning in populations of spiking neurons.
Nature neuroscience, 12(3):250–252.
Urbanowicz, R. J., Moore, J. H. (2009). Learning classifier systems: A complete introduction,
review, and roadmap. Journal of Artificial Evolution and Applications. 10.1155/2009/736398.
Valentin, V. V., Dickinson, A., O’Doherty, J. P. (2007). Determining the neural substrates ofgoal-directed learning in the human brain. The Journal of Neuroscience, 27(15):4019–4026.
van Hasselt, H. (2010). Double Q-learning. In Advances in Neural Information Processing
Systems 23, pp. 2613–2621. Curran Associates, Inc.
van Hasselt, H. (2011). Insights in Reinforcement Learning: Formal Analysis and Empirical
Evaluation of Temporal-di↵erence Learning. SIKS dissertation series number 2011-04.
van Hasselt, H. (2012). Reinforcement learning in continuous state and action spaces. In M.
Wiering and M. van Otterlo (Eds.), Reinforcement Learning: State-of-the-Art, pp. 207–251.
Springer-Verlag Berlin Heidelberg.
van Hasselt, H., Sutton, R. S. (2015). Learning to predict independent of span. ArXiv:1508.04582.
Van Roy, B., Bertsekas, D. P., Lee, Y., Tsitsiklis, J. N. (1997). A neuro-dynamic programming
approach to retailer inventory management. In Proceedings of the 36th IEEE Conference on
Decision and Control, Vol. 4, pp. 4052–4057.
van Seijen, H. (2011). Reinforcement Learning under Space and Time Constraints. University of
Amsterdam PhD thesis. Hague: TNO.
van Seijen, H. (2016). E↵ective multi-step temporal-di↵erence learning for non-linear function
approximation. ArXiv:1608.05151.
van Seijen, H., Sutton, R. S. (2013). Efficient planning in MDPs by small backups. In: Proceedingsof the 30th International Conference on Machine Learning, pp. 361–369.
van Seijen, H., Sutton, R. S. (2014). True online TD( ). In Proceedings of the 31st International
Conference on Machine Learning, pp. 692–700. JMLR W&CP 32(1),van Seijen, H., Mahmood, A. R., Pilarski, P. M., Machado, M. C., Sutton, R. S. (2016). True
online temporal-di↵erence learning. Journal of Machine Learning Research, 17 (145):1–40.
van Seijen, H., Van Hasselt, H., Whiteson, S., Wiering, M. (2009). A theoretical and empiricalanalysis of Expected Sarsa. In IEEE Symposium on Adaptive Dynamic Programming andReinforcement Learning, pp. 177–184.

516

References

van Seijen, H., Whiteson, S., van Hasselt, H., Wiering, M. (2011). Exploiting best-match
equations for efficient reinforcement learning. Journal of Machine Learning Research 12 :2045–
2094.
Varga, R. S. (1962). Matrix Iterative Analysis. Englewood Cli↵s, NJ: Prentice-Hall.
Vasilaki, E., Frémaux, N., Urbanczik, R., Senn, W., Gerstner, W. (2009). Spike-based reinforcement learning in continuous state and action space: when policy gradient methods fail.
PLoS Computational Biology, 5(12).
Viswanathan, R., Narendra, K. S. (1974). Games of stochastic automata. IEEE Transactions
on Systems, Man, and Cybernetics, 4(1):131–135.
Wagner, A. R. (2008). Evolution of an elemental theory of Pavlovian conditioning. Learning &Behavior, 36 (3):253–265.
Walter, W. G. (1950). An imitation of life. Scientific American, 182(5):42–45.
Walter, W. G. (1951). A machine that learns. Scientific American, 185(2):60–63.
Waltz, M. D., Fu, K. S. (1965). A heuristic approach to reinforcement learning control systems.
IEEE Transactions on Automatic Control, 10(4):390–398.
Watkins, C. J. C. H. (1989). Learning from Delayed Rewards. PhD thesis, University ofCambridge.
Watkins, C. J. C. H., Dayan, P. (1992). Q-learning. Machine Learning, 8(3-4):279–292.
Werbos, P. J. (1977). Advanced forecasting methods for global crisis warning and models ofintelligence. General Systems Yearbook, 22(12):25–38.
Werbos, P. J. (1982). Applications of advances in nonlinear sensitivity analysis. In R. F. Drenick
and F. Kozin (Eds.), System Modeling and Optimization, pp. 762–770. Springer-Verlag.
Werbos, P. J. (1987). Building and understanding adaptive systems: A statistical/numericalapproach to factory automation and brain research. IEEE Transactions on Systems, Man,
and Cybernetics, 17(1):7–20.
Werbos, P. J. (1988). Generalization of back propagation with applications to a recurrent gas
market model. Neural Networks, 1(4):339–356.
Werbos, P. J. (1989). Neural networks for control and system identification. In Proceedings ofthe 28th Conference on Decision and Control, pp. 260–265. IEEE Control Systems Society.
Werbos, P. J. (1992). Approximate dynamic programming for real-time control and neuralmodeling. In D. A. White and D. A. Sofge (Eds.), Handbook of Intelligent Control: Neural,
Fuzzy, and Adaptive Approaches, pp. 493–525. Van Nostrand Reinhold, New York.
Werbos, P. J. (1994). The Roots of Backpropagation: From Ordered Derivatives to Neural
Networks and Political Forecasting (Vol. 1). John Wiley and Sons.
Wiering, M., Van Otterlo, M. (2012). Reinforcement Learning: State-of-the-Art. Springer-Verlag
Berlin Heidelberg.
White, A. (2015). Developing a Predictive Approach to Knowledge. PhD thesis, University ofAlberta, Edmonton.
White, D. J. (1969). Dynamic Programming. Holden-Day, San Francisco.
White, D. J. (1985). Real applications of Markov decision processes. Interfaces, 15(6):73–83.
White, D. J. (1988). Further real applications of Markov decision processes. Interfaces,
18(5):55–61.
White, D. J. (1993). A survey of applications of Markov decision processes. Journal of the
Operational Research Society, 44(11):1073–1096.
White, A., White, M. (2016). Investigating practical linear temporal di↵erence learning. In
Proceedings of the 2016 International Conference on Autonomous Agents and Multiagent
Systems, pp. 494–502.

References

517

Whitehead, S. D., Ballard, D. H. (1991). Learning to perceive and act by trial and error.
Machine Learning, 7(1):45–83.
Whitt, W. (1978). Approximations of dynamic programs I. Mathematics of Operations Research,
3(3):231–243.
Whittle, P. (1982). Optimization over Time, vol. 1. Wiley, New York.
Whittle, P. (1983). Optimization over Time, vol. 2. Wiley, New York.
Wickens, J., Kötter, R. (1995). Cellular models of reinforcement. In J. C. Houk, J. L. Davis and
D. G. Beiser (Eds.), Models of Information Processing in the Basal Ganglia, pp. 187–214.
MIT Press, Cambridge, MA.
Widrow, B., Gupta, N. K., Maitra, S. (1973). Punish/reward: Learning with a critic in adaptive
threshold systems. IEEE Transactions on Systems, Man, and Cybernetics, 3(5):455–465.
Widrow, B., Ho↵, M. E. (1960). Adaptive switching circuits. In 1960 WESCON Convention
Record Part IV, pp. 96–104. Institute of Radio Engineers, New York. Reprinted in J. A.
Anderson and E. Rosenfeld, Neurocomputing: Foundations of Research, pp. 126–134. MIT
Press, Cambridge, MA, 1988.
Widrow, B., Smith, F. W. (1964). Pattern-recognizing control systems. In J. T. Tou andR. H. Wilcox (Eds.), Computer and Information Sciences, pp. 288–317. Spartan, Washington,
DC.
Widrow, B., Stearns, S. D. (1985). Adaptive Signal Processing. Prentice-Hall, Englewood Cli↵s,
NJ.
Wiener, N. (1964). God and Golem, Inc: A Comment on Certain Points where Cybernetics
Impinges on Religion. MIT Press, Cambridge, MA.
Wiewiora, E. (2003). Potential-based shaping and Q-value initialization are equivalent. Journalof Artificial Intelligence Research, 19 :205–208.
Williams, R. J. (1986). Reinforcement learning in connectionist networks: A mathematicalanalysis. Technical Report ICS 8605. Institute for Cognitive Science, University of California
at San Diego, La Jolla.
Williams, R. J. (1987). Reinforcement-learning connectionist systems. Technical Report
NU-CCS-87-3. College of Computer Science, Northeastern University, Boston.
Williams, R. J. (1988). On the use of backpropagation in associative reinforcement learning.
In Proceedings of the IEEE International Conference on Neural Networks, pp. I-263–I-270.
IEEE San Diego section and IEEE TAB Neural Network Committee.
Williams, R. J. (1992). Simple statistical gradient-following algorithms for connectionist
reinforcement learning. Machine Learning, 8(3-4):229–256.
Williams, R. J., Baird, L. C. (1990). A mathematical analysis of actor–critic architectures forlearning optimal controls through incremental dynamic programming. In Proceedings of the
Sixth Yale Workshop on Adaptive and Learning Systems, pp. 96–101. Center for Systems
Science, Dunham Laboratory, Yale University, New Haven.
Wilson, R. C., Takahashi, Y. K., Schoenbaum, G., Niv, Y. (2014). Orbitofrontal cortex as acognitive map of task space. Neuron, 81(2):267–279.
Wilson, S. W. (1994). ZCS, A zeroth order classifier system. Evolutionary Computation,2(1):1–18.
Wise, R. A. (2004). Dopamine, learning, and motivation. Nature Reviews Neuroscience,5(6):1–12.
Witten, I. H. (1976a). Learning to Control. University of Essex PhD thesis.
Witten, I. H. (1976b). The apparent conflict between estimation and control—A survey of the
two-armed problem. Journal of the Franklin Institute, 301(1-2):161–189.

518

References

Witten, I. H. (1977). An adaptive optimal controller for discrete-time Markov environments.
Information and Control, 34(4):286–295.
Witten, I. H., Corbin, M. J. (1973). Human operators and automatic adaptive controllers: A
comparative study on a particular control task. International Journal of Man–Machine
Studies, 5(1):75–104.
Woodbury, T., Dunn, C., and Valasek, J. (2014). Autonomous soaring using reinforcement
learning for trajectory generation. In 52nd Aerospace Sciences Meeting, p. 0990.
Woodworth, R. S. (1938). Experimental Psychology. New York: Henry Holt and Company.
Xie, X., Seung, H. S. (2004). Learning in neural networks by reinforcement of irregular spiking.
Physical Review E, 69(4):041909.
Xu, X., Xie, T., Hu, D., Lu, X. (2005). Kernel least-squares temporal di↵erence learning.
International Journal of Information Technology, 11 (9):54–63.
Yagishita, S., Hayashi-Takagi, A., Ellis-Davies, G. C. R., Urakubo, H., Ishii, S., Kasai, H. (2014).
A critical time window for dopamine actions on the structural plasticity of dendritic spines.
Science, 345(6204):1616–1619.
Yee, R. C., Saxena, S., Utgo↵, P. E., Barto, A. G. (1990). Explaining temporal di↵erences tocreate useful concepts for evaluating states. In Proceedings of the Eighth National Conferenceon Artificial Intelligence, pp. 882–888. AAAI Press, Menlo Park, CA.
Yin, H. H., Knowlton, B. J. (2006). The role of the basal ganglia in habit formation. Nature
Reviews Neuroscience, 7(6):464–476.
Young, P. (1984). Recursive Estimation and Time-Series Analysis. Springer-Verlag, Berlin.
Yu, H. (2010). Convergence of least squares temporal di↵erence methods under general conditions.
International Conference on Machine Learning 27, pp. 1207–1214.
Yu, H. (2012). Least squares temporal di↵erence methods: An analysis under general conditions.
SIAM Journal on Control and Optimization, 50 (6):3310–3343.
Yu, H. (2015). On convergence of emphatic temporal-di↵erence learning. In Proceedings of the
28th Annual Conference on Learning Theory, JMLR W&CP 40. Also ArXiv:1506.02582.
Yu, H. (2016). Weak convergence properties of constrained emphatic temporal-di↵erence learningwith constant and slowly diminishing stepsize. Journal of Machine Learning Research,
17 (220):1–58.
Yu, H. (2017). On convergence of some gradient-based temporal-di↵erences algorithms foro↵-policy learning. ArXiv:1712.09652.
Yu, H., Mahmood, A. R., Sutton, R. S. (2017). On generalized bellman equations and temporaldi↵erence learning. ArXiv:17041.04463. A summary appeared in Proceedings of the Canadian
Conference on Artificial Intelligence, pp. 3–14. Springer.

IndexPage numbers in italics are recommended to be consulted first. Page numbers in bold containboxed algorithms.
k-armed bandits, 25–45

for Sarsa, 129
for Expected Sarsa, 134
absorbing state, 57
for Sarsa( ), 304
access-control queuing example, 256
for TD( ), 289
action preferences, 322, 329, 336, 455
for Q( ), 313
in bandit problems, 37, 42
for Tree Backup( ), 314
action-value function, see value function, action
for Truncated TD( ), 296
action-value methods, 321
for n-step Q( ), 155for bandit problems, 27
for n-step Expected Sarsa, 146
actor–critic, 21, 239, 321, 331–332, 338, 406
for n-step Sarsa, 146
advantage, A2C, 338
for n-step TD, 142
one-step (episodic), 332
for n-step Tree Backup, 152with eligibility traces (episodic), 332
for Samuel’s Checker Player, 428
with eligibility traces (continuing), 333
compound, 288
neural, 395–415
half backups, 62
addiction, 409–410
backward view of eligibility traces, 288, 293
afterstates, 137, 140, 181, 182, 191, 424, 430
Baird’s counterexample, 261–264, 280, 283, 285
agent–environment interface, 47–58, 466
bandit algorithm, simple, 32
all-actions algorithm, 326
bandit problems, 25–45
AlphaGo, AlphaGo Zero, AlphaZero, 441–450
basal ganglia, 386
Andreae, John, 17, 21, 69, 89
baseline, 37–40, 329, 330, 338
ANN, see artificial neural networks
behavior policy, 103, 110, see o↵-policy learning
applications and case studies, 421–457
Bellman equation, 14
approximate dynamic programming, 15
for v⇡ , 59
artificial intelligence, xvii, 1, 472, 478
for q⇡ , 78
artificial neural networks, 223–228, 238–240,
for optimal value functions: v⇤ and q⇤ , 63
395–398, 423, 430, 436–450, 472
di↵erential, 250
associative reinforcement learning, 45, 418
for options, 463
associative search, 41
Bellman error, 268, 270, 272, 273
asynchronous dynamic programming, 85, 88
learnability of, 274–278
Atari video game play, 436–441
vector, 267–269
auxiliary tasks, 460–461, 468, 474
Bellmanoperator, 267–269, 286
average reward setting, 249–255, 258, 464
Bellman residual, 286, see Bellman error
averagers, 264
Bellman, Richard, 14, 71, 89, 241
binary features, 215, 222, 245, 304, 305
backgammon, 11, 21, 182, 184, 421–426
backpropagation, 21, 225–227, 239, 407, 424, bioreactor example, 51
blackjack example, 93–94, 99, 106
436, 439
blocking maze example, 166
backup diagram, 60, 139
for dynamic programming, 59, 61, 64, 172 bootstrapping, 89, 189, 308
n-step, 141–158, 255
for Monte Carlo methods, 94
and dynamic programming, 89
for Q-learning, 134
for TD(0), 121
and function approximation, 208, 264–274

520

and Monte Carlo methods, 95
and stability, 263–265
and TD learning, 120
assessment of, 124–128, 248, 264, 291, 318
in psychology, 345, 349, 354, 355
parameter ( or n), 291, 307, 399
BOXES, 18, 71, 237
branching factor, 173–177, 422
breakfast example, 5, 22
bucket-brigade algorithm, 19, 21, 139

Indexdeadly triad, 264
deep learning, 12, 223, 441, 472–474, 479
deep reinforcement learning, 236
deep residual learning, 227delayed reinforcement, 361–363
delayed reward, 1, 47, 249
dimensions of reinforcement learning methods,
189–191
direct and indirect RL, 162, 164, 192
discounting, 55, 199, 243, 249, 282, 324, 328,
427, 459
in pole balancing, 56
state dependent, 307
deprecated, 253, 256
distribution models, 159, 185
dopamine, 377, 381–387, 413–419
and addiction, 409–410
double learning, 134–136, 140
DP, see dynamic programming
driving-home example, 122–123
Dyna architecture, 164, 161–170
dynamic programming, 13–15, 73–90, 174, 262
and artificial intelligence, 89
and function approximation, 241
and options, 463
and the deadly triad, 264
computational efficiency of, 87

catastrophic interference, 472
certainty-equivalence estimate, 128
chess, 4, 20, 54, 182, 450
classical conditioning, 20, 343–357
blocking, 371
and higher-order conditioning, 345–355
delay and trace conditioning, 344
Rescorla-Wagner model, 346–349
TD model, 349–357
classifier systems, 19, 21
cli↵ walking example, 132, 133
CMAC, see tile codingcoarse coding, 215–220, 238
cognitive maps, 363–364
collective reinforcement learning, 404–407
complex backups, see compound update
compound stimulus, 345, 346–356, 371, 382
eligibility traces, 287–320, 350, 362, 398–403
compound update/backup, 288, 319
accumulating, 300, 306, 310
conditioned/unconditioned stimulus, conditioned
replacing, 301, 306
response (CS/US, CR), 344
dutch, 300–303
constant-↵ MC, 120
contingent/non-contingent, 399–403, 411
contextual bandits, 41o↵-policy, 309–316
continuing tasks, 54, 57, 70, 124, 249, 294
with state-dependent and , 309–316
continuous action, 73, 244, 335–336
Emphatic-TD methods, 234–235, 315
continuous state, 73, 223, 238o↵-policy, 281–282
continuous time, 11, 71
environment, 47–58
control and prediction, 342
episodes, episodic tasks, 11, 54–58, 91
control theory, 4, 71
error reduction property, 144, 288
control variates, 150–152, 155, 281
evaluative feedback, 17, 25, 47
and eligibility traces, 309–312
evolution, 7, 359, 374, 471
credit assignment, 11, 17, 19, 47, 294, 401
evolutionary methods, 7, 8, 9, 11, 19
in psychology, 346, 361
expected approximate value, 148, 155
structural, 385, 405, 407
Expected Sarsa, 133, see also Sarsa, Expected
critic, 18, 239, 346, 417, see actor–critic
expected update, 75, 172–181, 189
cumulant, 459
experience replay, 440–441
curiosity, 474
explore/exploit dilemma, 3, 103, 472
curse of dimensionality, 4, 14, 221, 231
cybernetics, xvii, 477
exploring starts, 96, 98–100, 178

Indexfeature construction, 210–223
final time step (T ), 54
Fourier basis, 211–215
function approximation, 195–200

521

n-step, 148–156
incremental implementationof averages, 30–33
of weighted averages, 109
instrumental conditioning, 357–361, see also
gambler’s example, 84
Law of E↵ectgame theory, 19
and motivation, 360–361
gazelle calf example, 5Thorndike’s puzzle boxes, 358
general value functions (GVFs), 459–463, 474 interest and emphasis, 234–235, 282, 316
generalized policy iteration (GPI), 86–87, 92, inverse reinforcement learning, 470
97, 138, 189
genetic algorithms, 19
Jack’s car rental example, 81–82, 137, 210
Gittins index, 43kernel-based function approximation, 232–233
gliding/soaring case study, 453–457
Klopf, A. Harry, xv, xvii, 19–21, 402–404, 411
goal, see reward signal
golf example, 61, 63, 66
latent learning, 192, 363, 366
gradient, 201
gradient descent, see stochastic gradient de- Law of E↵ect, 15–16, 45, 343, 358–361, 417
learning automata, 18
scent
Least Mean Square (LMS) algorithm, 279, 301
Gradient-TD methods, 278–281, 314–315
Least-Squares TD (LSTD), 228–229
greedy or "-greedy
linear function approx., 204–209, 266–269
as exploiting, 26–28
linear programming, 87, 90
as shortsighted, 64
local and global optima, 200"-greedy policies, 100
gridworld examples, 60, 65, 76, 147
Markov decision process (MDP), 2, 14, 47–71
cli↵ walking, 132
Markov property, 49, 115, 465–468
Dyna blocking maze, 166
Markov reward process (MRP), 125
Dyna maze, 164
maximization bias, 134–136
Dyna shortcut maze, 167
maximum-likelihood estimate, 128
windy, 130, 131
MC, see Monte Carlo methods
Mean Square
habitual and goal-directed control, 364–368
Bellman Error, BE, 268
hedonistic neurons, 402–404
Projected Bellman Error, PBE, 269
heuristic search, 181–183, 190
Return Error, RE, 275
as sequences of backups, 183
TD Error, TDE, 270
in Samuel’s checkers player, 426
Value Error, VE, 199–200
in TD-Gammon, 425
memory-based function approx., 230–232
history of reinforcement learning, 13–22
Michie, Donald, 17, 71, 117
Holland, John, 19, 21, 44, 139, 241
Minsky, Marvin, 16, 17, 20, 89
Hull, Clark, 16, 359, 360, 362–363
model of the environment, 7, 159
model-based and model-free methods, 7, 159
importance sampling, 103–117, 151, 257
in animal learning, 363–368
ratio, 104, 148, 258
model-based reinforcement learning, 159–193
weighted and ordinary, 105, 106
in neuroscience, 407–409
and eligibility traces, 309–312
Monte Carlo methods, 91–117
and infinite variance, 106
first- and every-visit MC, 92
discounting aware, 112–113
first-visit MC control, 101
incremental implementation, 109
per-decision, 114–115
first-visit MC prediction, 92

522

gradient method for v⇡ , 202Monte Carlo ES (Exploring Starts), 99o↵-policy control, 111, 110–112o↵-policy prediction, 103–109, 110
Monte Carlo Tree Search (MCTS), 185–188
motivation, 360–361
mountain car example, 244–248, 305, 306
multi-armed bandits, 25–45
n-step methods, 141–158
Q( ), 156
Sarsa, 147, 247
di↵erential, 255o↵-policy, 149
TD, 144
Tree Backup, 154
truncated -return, 295
naughts and crosses, see tic-tac-toe
neural networks, see artificial neural networks
neurodynamic programming, 15
neuroeconomics, 413, 419
neuroscience, 4, 21, 377–419
nonstationarity, 30, 32–36, 44, 255
inherent, 91, 198
notation, xiii, xix

Index

approximate
control, 244, 247, 251, 255
prediction, 202, 203, 209Monte Carlo, 101, 100–103, 328, 330
n-step, 144, 147
Sarsa, 130, 129–131
TD(0), 120, 119–128
with eligibility traces, 293, 300, 305, 307
operant conditioning, see instrumental learning
optimal control, 2, 14–15, 21
optimistic initial values, 34–35, 192
optimizing memory control, 432–436
options, 461–464
models of, 462

pain and pleasure, 6, 16, 413
Partially Observable MDPs (POMDPs), 467
Pavlov, Ivan, 16, 343–345, 362
Pavlovianconditioning, see classical conditioning
control, 343, 371, 373, 478
personalizing web services, 450–453
planning, 3, 5, 7, 11, 138, 159–193
in psychology, 363, 364, 366
with learned models, 161–168, 473
with options, 461, 463
observations, 464
policy, 6, 41, 58
o↵-policy methods, 257–286
hierarchical, 462
vs on-policy methods, 100, 103
soft and "-soft, 100–103, 110
Monte Carlo, 103–115
policy approximation, 321–324
Q-learning, 131
policy evaluation, 74–76, see also prediction
Expected Sarsa, 133–134
iterative, 75
n-step, 148–156
policy gradient methods, 321–338
n-step Q( ), 156
REINFORCE, 328, 330
n-step Sarsa, 149
actor–critic, 332, 333
policy gradient theorem, 324–326
n-step Tree Backup, 154
proof, episodic case, 325
and eligibility traces, 309–316
proof, continuing case, 334
Emphatic-TD( ), 315
policy improvement, 76–80
GQ( ), 315
theorem, 78, 101
GTD( ), 314
policy iteration, 14, 80, 80–82
HTD( ), 315
polynomial basis, 210–211
Q( ), 312–314
prediction, 74–76, see also policy evaluation
Tree Backup( ), 312–314
and control, 342
reducing variance, 283–284
Monte Carlo, 92–97
on-policy distribution, 175, 199, 208, 258, 262,
281, 282o↵-policy, 103–108
vs uniform distribution, 176
TD, 119–126
on-policy methods, 100with approximation, 197–242
actor–critic, 332, 333
prior knowledge, 12, 34, 54, 137, 236, 324, 471

Indexprioritized sweeping, 170, 168–171
projected Bellman error, 285
vector, 267, 269
proximal TD methods, 286
pseudo termination, 282, 308
psychology, 4, 13, 19, 20, 341–376
Q( ), Watkins’s, 312–314
Q-function, see action-value function
Q-learning, 21, 131, 131–135
double, 136
Q-planning, 161
Q( ), 156, 154–156
queuing example, 252523

and reinforcement, 373–375, 380–381
design of, 469–472, 477
intrinsic, 474
sparse, 469–470
rod maneuvering example, 171
rollout algorithms, 183–185
root mean square (RMS) error, 125safety, 434, 478
sample and expected updates, 121, 170–174
sample or simulation model, 115
sample-average method, 27
Samuel’s checkers player, 20, 241, 426–429
Sarsa, 130, 129–131, 244
vs Q-learning, 132
R-learning, 256di↵erential, one-step, 251
racetrack exercise, 111
Expected, 133–134, 140
radial basis functions (RBFs), 221–222
n-step, 148
random walk, 95
n-step o↵-policy, 150
5-state, 125, 126, 127
double, 136
19-state, 144, 291
n-step, 147, 145–148, 247
TD( ) results on, 294, 295, 299
di↵erential, 255
1000-state, 203–209, 217, 218o↵-policy, 149
Fourier and polynomial bases, 214
Sarsa( ), 305, 303–307
real-time dynamic programming, 177–180
true online, 307
recycling robot example, 52
Schultz, Wolfram, 387–395, 410
REINFORCE, 328, 326–331
search control, 163
with baseline, 330
secondary reinforcement, 20, 346, 354, 369
reinforcement learning, 1–22
selective bootstrap adaptation, 239
reinforcement signal, 380
semi-gradient methods, 202, 258–259
representation learning, 473
SGD, see stochastic gradient descentresidual-gradient algorithm, 272–274, 277
Shannon, Claude, 16, 20, 71, 426
naive, 270, 271
shaping, 360, 470
return, 54–58
Skinner, B. F., 359–360, 375, 470, 479
n-step, 143soap bubble example, 95
soft and "-soft policies, 100–103, 110
for Q( ), 155
soft-max, 322–323, 329, 336, 400, 445, 455
for action values, 146
for bandits, 37, 45
for Expected Sarsa, 148
spike-timing-dependent plasticity (STDP), 401
for Tree Backup, 153
state, 7, 48, 49
with control variates, 150, 151
kth-order history approach, 468
with function approximation, 209
and observations, 464–468
di↵erential, 250, 255, 334
Markov property, 465–468
flat partial, 113
belief, 467
with state-dependent termination, 308
latent, 467-return, 288–291
observable operator models (OOMs), 467
truncated, 296
partially observable MDPs, 14, 467
reward prediction error hypothesis, 381–383,
387–395
predictive state representations, 467
reward signal, 1, 6, 48, 53, 361, 380, 383, 397
state-update function, 465

524

state aggregation, 203–204
state-update function, 465
step-size parameter, 10, 31–33, 120, 125, 126
automatic adaptation, 238
in DQN, 439, 440
in psychological models, 347, 348
selecting manually, 222–223
with coarse coding, 216with Fourier features, 213with tile coding, 217, 223
stochastic approx. convergence conditions, 33
stochastic gradient descent (SGD), 200–204
in the Bellman error, 269–278
strong and weak methods, 4supervised learning, xvii, 2, 17–19, 198
sweeps, 75, 160, see also prioritized sweeping
synaptic plasticity, 379
Hebbian, 400
two-factor and three factor, 400
system identification, 364
tabular solution methods, 23
targetpolicy, 103, 110
of update, 31, 143, 198
TD, see temporal-di↵erence learning
TD error, 121n-step, 255di↵erential, 250
with function approximation, 270
TD( ), 293, 292–295
truncated, 295–297
true online, 300, 299–301
TD-Gammon, 21, 421–426
temporal abstraction, 461–464
temporal-di↵erence learning, 10, 119–140
history of, 20–21
advantages of, 124–126
optimality of, 126–128
TD(0), 120, 203
TD(1), 294
TD( ), 293, 292–295
true online, 300, 299–301-return methodso↵-line, 290
online, 297–299
n-step, 144, 141–158, 209
termination function, 307, 459
Thompson sampling, 43, 45

Index

Thorndike, Edward, see Law of E↵ect
tic-tac-toe, 8–13, 17, 137
tile coding, 217–221, 223, 238, 246, 434, 435
Tolman, Edward, 364, 408
trace-decay parameter ( ), 287, 289, 290, 292
state dependent, 307
trajectory sampling, 174–177
transition probabilities, 49
Tree Backup
n-step, 152–153, 154
Tree-Backup( ), 312–314
trial-and-error, 1, 7, 15–21, 403, 404, see also
instrumental conditioning
true online TD( ), 300, 299–301
Tsitsiklis and Van Roy’s Counterexample, 263
undiscounted continuing tasks, see average reward setting
unsupervised learning, 2, 226
value, 6, 26, 47
value function, 6, 58–67
for a given policy: v⇡ and q⇡ , 58
for an optimal policy: v⇤ and q⇤ , 62action, 58, 63, 65, 71, 129, 131approximate action values: q̂(s, a, w), 243
approximate state values: v̂(s,w), 197di↵erential, 243
vs evolutionary methods, 11
value iteration, 83, 82–84
value-function approximation, 198
Watkins, Chris, 15, 21, 89, 320
Watson (Jeopardy! player), 429–432
Werbos, Paul, 14, 21, 70, 89, 139, 239
Witten, Ian, 21, 70

525

Adaptive Computation and Machine Learning

Francis Bach, Editor

Bioinformatics: The Machine Learning Approach, Pierre Baldi and Søren Brunak
Reinforcement Learning: An Introduction, Richard S. Sutton and Andrew G. Barto
Graphical Models for Machine Learning and Digital Communication, Brendan J. Frey
Learning in Graphical Models, Michael I. Jordan
Causation, Prediction, and Search, second edition, Peter Spirtes, Clark Glymour, andRichard Scheines
Principles of Data Mining, David Hand, Heikki Mannila, and Padhraic Smyth
Bioinformatics: The Machine Learning Approach, second edition, Pierre Baldi and Søren
BrunakLearning Kernel Classifiers: Theory and Algorithms, Ralf Herbrich
Learning with Kernels: Support Vector Machines, Regularization, Optimization, andBeyond, Bernhard Schölkopf and Alexander J. Smola
Introduction to Machine Learning, Ethem Alpaydin
Gaussian Processes for Machine Learning, Carl Edward Rasmussen and Christopher K.I.
Williams
Semi-Supervised Learning, Olivier Chapelle, Bernhard Schölkopf, and Alexander Zien,
Eds.
The Minimum Description Length Principle, Peter D. Grünwald
Introduction to Statistical Relational Learning, Lise Getoor and Ben Taskar, Eds.

526

Probabilistic Graphical Models: Principles and Techniques, Daphne Koller and Nir Friedman
Introduction to Machine Learning, second edition, Ethem Alpaydin
Machine Learning in Non-Stationary Environments: Introduction to Covariate Shift
Adaptation, Masashi Sugiyama and Motoaki Kawanabe
Boosting: Foundations and Algorithms, Robert E. Schapire and Yoav Freund
Machine Learning: A Probabilistic Perspective, Kevin P. Murphy
Foundations of Machine Learning, Mehryar Mohri, Afshin Rostami, and Ameet Talwalker
Introduction to Machine Learning, third edition, Ethem Alpaydin
Deep Learning, Ian Goodfellow, Yoshua Bengio, and Aaron Courville
Elements of Causal Inference, Jonas Peters, Dominik Janzing, and Bernhard Schölkopf
Machine Learning for Data Streams, with Practical Examples in MOA, Albert Bifet,
Ricard Gavaldà, Geo↵rey Holmes, Bernhard Pfahringer