# Making a Difference Tempore: Insights from 'Reinforcement Learning: An Introduction'

> The safety of artificial intelligence applications involving reinforcement learning is a topic that deserves careful attention.

Foreword
========

Let's get down to business.

Reinforcement Learning
======================

1: Introduction
---------------

2: Multi-armed Bandits
----------------------

_Bandit basics, including nonstationarity, the value of optimism for incentivizing exploration, and upper-confidence-bound action selection._

Some explore/exploit results are relevant to daily life - I highly recommend reading _[Algorithms to Live By: The Computer Science of Human Decisions](https://www.amazon.com/Algorithms-Live-Computer-Science-Decisions/dp/1627790365)_.

3: Finite Markov Decision Processes
-----------------------------------

_The framework._

4: Dynamic Programming
----------------------

_Policy evaluation, policy improvement, policy iteration, value iteration, generalized policy iteration. What a nomenclative nightmare._

5: Monte Carlo Methods
----------------------

_Prediction, control, and importance sampling._

### Importance Sampling

After gathering data with our behavior policy < refresh to render LaTeX > , we then want to approximate the value function for the target policy < refresh to render LaTeX > . In off-policy methods, the policy we use to gather the data is different from the one whose value < refresh to render LaTeX > we're trying to learn; in other words, the distribution of states we sample is different. This gives us a skewed picture of < refresh to render LaTeX > , so we must overcome this bias.

If < refresh to render LaTeX > can take all of the actions that < refresh to render LaTeX > can (_i.e._, < refresh to render LaTeX > ), we can overcome by adjusting the return < refresh to render LaTeX > of taking a series of actions < refresh to render LaTeX > using the importance-sampling ratio < refresh to render LaTeX > . This cleanly recovers < refresh to render LaTeX > by the definition of expectation:

<refresh to render LaTeX>

Then after observing some set of returns (where < refresh to render LaTeX > are the relevant returns for state < refresh to render LaTeX > ), we define the state's value as the average adjusted observed return

<refresh to render LaTeX>

However, the < refresh to render LaTeX > 's can be arbitrarily large (suppose < refresh to render LaTeX > and < refresh to render LaTeX > ; < refresh to render LaTeX > ), so the variance of this estimator can get pretty big. To get an estimator whose variance converges to 0, try

<refresh to render LaTeX>

### Death and Discounting

Instead of viewing the discount factor < refresh to render LaTeX > as measuring how much we care about the future, think of it as encoding the probability we don't terminate on any given step. That is, we expect with < refresh to render LaTeX > probability to die on the next turn, so we discount rewards accordingly.

This intuition hugs pre-theoretic understanding much more closely; if you have just 80 years to live, you might dive that big blue sky. If, however, you imagine there's a non-trivial chance that humanity can be more than just a flash in the pan, you probably care more about the far future.

6: Temporal-Difference Learning
-------------------------------

_The tabular triple threat:_ < refresh to render LaTeX > _,_ < refresh to render LaTeX > _, and_ < refresh to render LaTeX > _-learning._

### Learning TD Learning

<refresh to render LaTeX>

### < refresh to render LaTeX >

*   < refresh to render LaTeX > is one-step bootstrapping of _state values_ < refresh to render LaTeX > . It helps you learn the value of a given policy, and is not used for control.
*   < refresh to render LaTeX > is on-policy one-step bootstrapping  of _action values_ < refresh to render LaTeX > using the quintuples < refresh to render LaTeX > .

> As in all on-policy methods, we continually estimate < refresh to render LaTeX > for the policy < refresh to render LaTeX > , and at the same time change < refresh to render LaTeX > toward greediness with respect to < refresh to render LaTeX > .

*   < refresh to render LaTeX > -learning is _off-policy_ one-step bootstrapping of action values < refresh to render LaTeX > .

*   You take an action using < refresh to render LaTeX > and then use the maximal action value at the next state in your TD target term.

### < refresh to render LaTeX > imization bias

With great branching factors come great biases, and optimistic bias is problematic for < refresh to render LaTeX > -learning.

Refresher: the < refresh to render LaTeX > -learning update rule for state < refresh to render LaTeX > , action < refresh to render LaTeX > , and new state < refresh to render LaTeX > is

<refresh to render LaTeX>

Suppose the rewards for all 100 actions at < refresh to render LaTeX > are distributed according to < refresh to render LaTeX > . All of these actions have a true (expected) value of 0, but the probability that none of their estimates are greater than .8 after 1 draw each is < refresh to render LaTeX > . The more actions you have, the higher the probability is that at the maximum is just an outlier. See: [regression toward the mean and mutual funds](https://en.wikipedia.org/wiki/Regression_toward_the_mean#Importance).

To deal with this, we toss another < refresh to render LaTeX > -learner into the mix; at any given update, one has their value updated and the other greedifies the policy. The double < refresh to render LaTeX > -learning scheme works because both < refresh to render LaTeX > -learners are unlikely to be biased in the same way. For some reason, this [wasn't discovered until 2010](https://papers.nips.cc/paper/3964-double-q-learning).

7: < refresh to render LaTeX > -step Bootstrapping
--------------------------------------------------

< refresh to render LaTeX > _-step everything._

8: Planning and Learning with Tabular Methods
---------------------------------------------

_Models, prioritized sweeping, expected vs. sample updates, MCTS, and rollout algorithms._

### Roles Models Play

_Distribution_ models include the full range of possible futures and their probabilities. For example, a distribution model for two fair coins: < refresh to render LaTeX > .

_Sample_ models just let you, well, sample. < refresh to render LaTeX > ... You could sample thousands of times and gain a high degree of confidence that the above distribution model is generating the data, but this wouldn't be your _only_ hypothesis (granted, it might have the lowest Kolmogorov complexity).

Note that a distribution model also can function as a sample model.

9: On-policy Prediction with Approximation
------------------------------------------

_We finally hit the_ good stuff_: value-function approximators and stochastic-/semi-gradient methods._

10: On-policy Control with Approximation
----------------------------------------

11: Off-policy Methods with Approximation
-----------------------------------------

_The_ < refresh to render LaTeX > _of function approximation, bootstrapping, and off-policy training converge to sow d̴iv̢erge͏nc̸e͟ and i͟nstab̕il̡i̡t̷y in your methods. Metal._

12: Eligibility Traces
----------------------

_I was told to skip this chapter; the first half was my favorite part of the book._

< refresh to render LaTeX > uses a backwards-view eligibility trace to update features, which I find elegant. Suppose you have some feature extraction function < refresh to render LaTeX > . Then you apply your TD update not only based on the features relevant at the current state, but also to the time-decaying traces of the features of previously-visited states. < refresh to render LaTeX > sets how quickly this eligibility decay happens; < refresh to render LaTeX > recovers < refresh to render LaTeX > , while < refresh to render LaTeX > recovers Monte-Carlo return methods.

When I was a kid, there was a museum I loved going to - it had all kinds of wacky interactive devices for kids. One of them took sounds and "held them in the air" at a volume which slowly decreased as a function of how long ago the sound was made. State features are sounds, and volume is eligibility.

13: Policy Gradient Methods
---------------------------

_The policy gradient theorem, REINFORCE, and actor-critic._

14: Psychology
--------------

_Creating a partial mapping between reinforcement learning and psychology._

### Mental, Territorial

There was a word I was looking for that "mental model" didn't quite seem to fit: "the model with respect to which we mentally simulate courses of action". CFAR's "inner sim" terminology didn't quite map either, as to me, that points to the _system-in-motion_ more than _that-on-which-the-system-runs_. The literature dubs this a cognitive map.

> Individual place cells within the hippocampus correspond to separate locations in the environment with the sum of all cells contributing to a single map of an entire environment. The strength of the connections between the cells represents the distances between them in the actual environment. The same cells can be used for constructing several environments, though individual cells' relationships to each other may differ on a map by map basis. The possible involvement of place cells in cognitive mapping has been seen in a number of mammalian species, including rats and macaque monkeys. Additionally, in a study of rats by Manns and Eichenbaum, pyramidal cells from within the hippocampus were also involved in representing object location and object identity, indicating their involvement in the creation of cognitive maps.

> Wikipedia, _[Cognitive map](https://en.wikipedia.org/wiki/Cognitive_map)_

In light of [my work on whitelisting](https://www.lesswrong.com/posts/H7KB44oKoSjSCkpzL/worrying-about-the-vase-whitelisting), I've been thinking about why we're so "object-oriented" in our mental lives. An off-the-cuff hypothesis: to better integrate with the rest of our mental models, the visual system directly links up to our object schema. One such object is then recognized and engraved as a discrete "thing" in our map. Hence, we emotionally "know" that the world "really is" made up of objects, and isn't just a collection of particles.

> Most of the information used by people for the cognitive mapping of spaces is gathered through the visual channel (Lynch, 1960).

[Lahav and Mioduser's research](http://playpen.icomtek.csir.co.za/~acdc/assistive%20devices/Artabilitation2008/archive/2004/papers/2004_S04_N4.pdf) somewhat supports this idea, suggesting that blind people not only have lower-fidelity and more declarative (as opposed to procedural / interactive) cognitive maps, they're also less likely to provide object-to-object descriptions.

_Epistemic status: I made a not-obviously-wrong hypothesis and found two pieces of corroborating evidence. If anyone who knows more about this wants to chime in, please do!_

15: Neuroscience
----------------

_The reward prediction error hypothesis, dopamine, neural actor-critic, hedonistic neurons, and addiction._

16: Applications and Case Studies
---------------------------------

_From checkers to checkmate._

17: Frontiers
-------------

_Temporal abstraction, designing reward signals, and the future of reinforcement learning. In particular, the idea I had for having a whitelist-enabled agent predict expected object-level transitions is actually one of the frontiers: general value functions. Rad._

### Pandora's AI Boxing

> The rapid pace of advances in AI has led to warnings that AI poses serious threats to our societies, even to humanity itself.

This chapter talks a fair amount about the serious challenges in AI alignment (not sure if you all have heard of that problem), which is heartening.

> As to safety, hazards possible with reinforcement learning are not completely different from those that have been managed successfully for related applications of optimization and control methods.

I'm not sure about that. Admittedly, I'm not as familiar with those solutions, but the challenge here seems to be of a qualitatively different caliber. Conditional on true AI's achievement, we'd want to have extremely high confidence that it pans out _before_ we flip the switch. The authors acknowledge that

> it may be impossible for the agent to achieve the designer's goal no matter what its reward signal is.

I don't think it's impossible, but it's going to be extremely hard to get formal probabilistic guarantees. I mean, if you don't know an agent's rationality, [you can't learn their utility function](https://www.lesswrong.com/posts/rtphbZbMHTLCepd6d/humans-can-be-assigned-any-values-whatsoever). If you do know their rationality but not their probability distribution over outcomes, [you still  can't learn their utility function](https://www.lesswrong.com/posts/kYgWmKJnqq8QkbjFj/bayesian-utility-representing-preference-by-probability).

This is just _one_ of the [many](https://arbital.com/p/advanced_safety/) [open problems in alignment theory](https://arbital.com/p/taskagi_open_problems). If that's not enough, there are always the unknown unknowns...

Final Thoughts
--------------

I read the "nearly-complete" draft of the second edition, available [here](http://incompleteideas.net/book/the-book-2nd.html). It was pretty good, but I did find most of the exercises either too easy or requiring considerable programming investment to set up the environment described. The former makes sense, as I've already taken a course on this, and I'm probably a bit above the introductory level.

Some graphs could have been more clearly designed, and some math in the proof of Linear < refresh to render LaTeX > 's convergence (p. 206-207) is underspecified:

> In general, < refresh to render LaTeX > will be reduced toward zero whenever < refresh to render LaTeX > is positive definite, meaning < refresh to render LaTeX > for real vector < refresh to render LaTeX > .

An additional precondition: < refresh to render LaTeX > can't be the zero vector.

> For a key matrix of this type, positive definiteness is assured if all of its columns sum to a nonnegative number.

Unless I totally missed what "this type" entails, this is false if taken at face value:

<refresh to render LaTeX>

has nonnegative column sums and is also indefinite, having eigenvalues of 3 and -7.

However, the claim _is_ true in the subtle way they use it - they're _actually_ showing that since the matrix is symmetric, real, and strictly diagonally dominant with positive diagonal entries, it's also positive definite. This could be made clearer.

In all, reading this book was definitely a positive experience.

Forwards
========

I'll be finishing _Analysis II_ before moving on to Jaynes's _Probability Theory_ in preparation for a class in the fall.

Dota 2
------

Recently, OpenAI [made waves with their _OpenAI Five_ Dota 2 bot](https://blog.openai.com/openai-five/). To REINFORCE what I just learned and solidified, I might make a post in the near future breaking down how _Five_ differs from the _Alpha(Go) Zero_ approach, quantifying my expectations for _The International_ for [calibration](https://en.wikipedia.org/wiki/Brier_score).

No Longer a Spectator
---------------------

Four months and one week ago, [I started my journey](https://www.lesswrong.com/s/KGYLvTqFiFE2CpHfJ/p/WPtdQ3JnoRSci87Dz) through the MIRI reading list. In those dark days, attempting a proof induced a stupor similar to that I encountered approaching a crush in grade school, my words and thoughts leaving me.

Six textbooks later and with a _little_ effort, I'm able to prove things like the convergence of Monte Carlo integration to the Riemann integral, threading together lessons from _All of Statistics_ and _Analysis I_;  target in mind, words and thoughts remaining firmly by my side.

> The rapid pace of advances in artificial intelligence has led to warnings that artificial intelligence poses serious threats to our societies, even to humanity itself. The renowned scientist and artificial intelligence pioneer Herbert Simon anticipated the warnings we are hearing today...

> He spoke of the eternal conflict between the promise and perils of any new knowledge, reminding us of the Greek myths of Prometheus, the hero of modern science, who stole fire from the gods for the benefit of mankind, and Pandora, whose box could be opened by a small and innocent action to release untold perils on the world.

> Simon urged us to recognize that as designers of our future and _not mere spectators_, the decisions _we_ make can tilt the scale in Prometheus' favor.

  

* * *

_If you are interested in working together to learn MIRI-relevant math, if you have a burning desire to knock the alignment problem down a peg, if you're in a_ scale-tilting _mood - I would be more than happy to work with you. [Messaging me](https://www.lesswrong.com/users/turntrout) may also net you an invitation to the MIRIx Discord server._