Dynamic Programming. This is part 4 of the RL tutorial…
By: Sagi Shaier
Tags: Rl Tutorial Series, Data Science, Reinforcement Learning Tutorial Series, Machine Learning, Reinforcement Learning

Chapter 4 — Dynamic Programming

The key concepts of this chapter:

- Generalized Policy Iteration (GPI)

- In place dynamic programming (DP)

- Asynchronous dynamic programming

Starting in this chapter, the assumption is that the environment is a finite Markov Decision Process (finite MDP).

As mentioned in the previous chapter, we can find the optimal policy once we found the optimal value functions, v∗ or q∗

Optimal value functions [ref]

In this chapter we’ll see how we can use DP algorithms to compute the value functions in a slightly different, less intractable way.

The general idea is to take these 2 equations, and turn them into update rules for for improving the approximations of our value functions.

It will make more sense later on.

Policy Evaluation

Policy evaluation means computing the state-value function Vπ for an arbitrary policy π.

The book refers to it as the “prediction problem”.

Bellman equation for Vπ [ref]

where π(a|s) is the probability of taking action “a” in state “s” under policy π.

This is similar to what we saw in the previous chapter.

Consider a sequence of approximate value functions v0, v1, v2, . . .

The initial approximation, v0, is chosen randomly.

Each following approximation is obtained by using the Bellman equation as an update rule:

This algorithm is called iterative policy evaluation.

It basically takes the old value of a state, and replace it with a new value obtained as shown in the equation.

The book refers to this kind of operation a full backup. Since in each iteration, we back up the value of every state once to produce the new approximate value function V(k+1).

Iterative policy evaluation algorithm [ref]

Policy Improvement

Consider some policy π. Would it be better or worse to change it to new policy?

One way to answer this is by taking some action “A” in some state “S”, and then to continue following our policy π.

The idea behind policy improvement is that if it’s better to take action “A” in state “S” once than to follow π all the time — then it is better to select action “A” every time, and that the new policy would in fact be a better overall.

From here it’s natural to consider changes in all states and to all possible actions rather than just a single state to a particular action.

For example, consider the new greedy policy π’:

greedy policy π’ [ref]

This greedy policy π’ takes the action that looks the best (argmax) after one step of lookahead (only the following states), according to Vπ.

This process of taking an old policy, and making a new & improved one by selecting greedy actions with respect to the value function of the original policy, is called policy improvement.

So far in this chapter we only considered deterministic policies — where given a state, the policy gives us an action. But in a stochastic policy π, instead of just getting a single action per state, we are given probabilities, π(a|s), for taking each action, a, in each state, s.

Not to worry though, policy improvement works with in the stochastic case as well.

Policy Iteration

Once a policy π to yield a better policy π’, we can then follow the same steps and improve it again to yield an even better π’’.

This looks like this:

Policy iteration [ref]

Where “E” denotes a policy evaluation, and “I” denotes a policy improvement.

Policy iteration [ref]

This technique of obtaining an optimal policy is called policy iteration

Value Iteration

One disadvantage of policy iteration is that in each iteration we have to go through a policy evaluation. Which in itself is an iterative computation going through the states.

Nevertheless, we can truncate the policy evaluation step in several ways.

As an example, we can stop the evaluation part after just one sweep (one backup of each state).

That algorithm is called value iteration

Note that this is similar to the policy evaluation backup, except for the fact that it requires the max to be taken over all actions.

Value iteration [ref]

Asynchronous Dynamic Programming

A disadvantage of the DP methods is that they require operating over the all the state. If our environment has lots of states, this can be computational expensive.

Asynchronous DP algorithms refers to the idea of updating the value estimates of only a subset of states rather than the entire set of states at every policy evaluation iteration. They simply update the values of states in any order, using the values of other states that are available.

Generalized Policy Iteration

As we’ve seen, policy iteration consists of policy evaluation (in which we make the value function consistent with the current policy) and policy improvement (in which we select greedy actions in our policy).

Generalized policy iteration (GPI) refers to the idea of letting these two processes — policy evaluation and policy improvement, to interact.

GPI works as follows:

We randomly initialize our value function estimates of every state, and start with a random policy.

We then evaluate the values of every state using this policy.

And we update the policy by making greedy action choices, with respect to the value functions (i.e. taking the action that moves you to the state with the highest value).

This can be seen in the following image:

GPI [ref]

GPI loops around until it converges to the optimal policy:

GPI convergence [ref]

Example:

Consider the following environment:

Example Env. [ref]

Here, we receive -1 reward on every state except for the gray boxes.

Starting GPI, we initialize the policy to have random actions (up, left, right, down):

Random Init. [ref]

And then we initialize the value estimates by putting -1 everywhere but the gray boxes:

Then, we simply take the greedy action from each state:

Moving towards state with higher values [ref]

Now we can use the Bellman equations to reestimate the value of each state:

And we repeat this until we reach the optimal policy:

GPI example [ref]

Until next time,

Cheers.