Policy and Value Iteration. An Introduction to Reinforcement... | by Steve Roberts | Towards Data Science

Baby Robot has been steadily making his way back to his mum. So far he has:
Overcome the Bandit Problem to recharge his batteries
Solved the Prediction Problem to evaluate the relative value of states in an environment, letting him escape from a simple grid level.
Seen how an environment can be represented as a Markov Decision Process (MDP) and evaluated using the Bellman Equations.
In this next instalment we’ll consider the Control Problem and investigate how the optimal policy can be found from estimates of the environment’s state values.
The associated Jupyter Notebook for this article can be found on Github. This contains all of the code required to setup and run the levels and algorithms described below.
Alternatively you can run the Jupyter Notebook for this article on Binder:
mybinder.org
Baby Robot has found himself in a very small room, so escaping from this shouldn’t be too much of a challenge. The only downside is that the roof has been leaking and, as we saw in the previous part when we looked at MDPs, Baby Robot doesn’t like the wet. Puddles slow him down and potentially cause him to skid.
We defined that:
Dry Squares:
Take one unit of time to move through, so a reward of -1 is received when moving to a square with no puddle.
Baby Robot won’t skid if there’s no puddle, so the target state will always be reached.
Small Puddles:
Take twice as long to move through, so stepping in one of these gives a reward of -2.
The probability of skidding when in a small puddle is 0.4, so 0.6 of the time you’ll reach the target state, the rest of the time you’ll skid and end up in one of the other possible states.
Large Puddles:
Take 4 times as long to move through, so a reward of -4 is given when entering a square containing a large puddle.
There’s an increased chance of skidding, so you’ll only reach the target with a 0.4 probability. Therefore there’s a 0.6 probability you’ll end up in one of the other possible states.
This makes the problem slightly more complex, since it’s no longer just a case of choosing which state you’d like to move to and getting a fixed reward. Now, when an action is chosen, Baby Robot may end up moving to a state that wasn’t the target state and instead receives the reward for the state where he ends up.
When a skid does occur the resultant state is decided by dividing the total probability of skidding by the number of possible non-target states. So, for the example shown in Figure 2 above, where Baby Robot is currently sitting in a large puddle and wanting to move East, towards the exit, the following next states and rewards are possible:
For the simple grid level, since we have complete information about the system, knowing all of its transition probabilities and rewards, we can use Dynamic Programming to turn the Bellman Equations into update functions. These can then be applied iteratively to calculate the state values.
The Bellman Expectation equation, giving the value of state ‘s’ when following policy ‘π’ is given by:
where:
π(a|s) is the probability of taking action a in state s.p(s’,r|s,a) is the probability of moving to the next state s’ and getting reward r when starting in state s and taking action a.r is the reward received after taking this action.γ is the discount factor.v(s’) is the value of the next state.
The value of a state is given by the sum over the probability of all actions, times the sum over the probability of the next state and reward, multiplied by the reward obtained and the discounted value of the state where you end up.
And as an update equation this can be written:
The only difference here is that the value of the state ‘s’, at iteration ‘k+1’, is calculated using the values of the next states that were already calculated at the previous iteration ‘k’. So the new estimate of a state’s value is derived from the previous estimates. When one estimate is based on another this is referred to as Bootstrapping.
In the example shown in Figure 2 above, where Baby Robot would like to choose the action to move East from the center square, we’re using a deterministic policy. In other words, for each state our policy will explicitly define which action should be chosen. Therefore there’ll be a single action, chosen with probability 1, so the sum over the actions in the above equations will disappear.
Similarly, when using iterative policy evaluation, it’s common to start with all the initial state value estimates set to zero, so the discounted value of the next states will also disappear for the first iteration.
As a result, for the state at the center of the grid level, where Baby Robot currently finds himself, the first estimate of the state’s value can be given by summing over the probabilities and rewards that are shown in the table above. So, for a policy that selects the East action in this state, the first estimate of the value of this state is:
If, instead of selecting the East action in this state, the policy defined that actions of North, South or West should be taken, then the first estimates of the state value would be given by:
From these first state estimates, where no future rewards are considered (since all next state values are currently zero), a policy that chooses to move West has the least negative state value estimate. This is due to its target state being a dry square, which gives the least negative immediate reward from the possible states. This estimate doesn’t yet contain any information about future rewards, which are required to let us find the best path through the grid. At the minute the estimated state value only the gives information that will keep Baby Robot the driest!
The state value estimates shown above are the values calculated for a deterministic policy, where a set action is defined for a state. If we instead used a stochastic policy, where actions are chosen probabilistically, then each of these state value estimates would simply be multiplied by the probability of that action being taken (the π(a|s) part in the Bellman equation) and then we’d add the results.
For the center square, with an equal probability of selecting any action (so each has a 0.25 probability of being chosen), the first estimated state value would be:
In all cases this estimated value is a measure of how good it is to be in this state under the current policy. To find which are the best states to be in, which in turn will let us find the best route through the grid, we need to repeat our calculation for all other states in the grid (commonly referred to as doing a sweep of all states). For the stochastic policy, where all possible actions in a state are equally likely, this gives us the state value estimates shown below:
Now that we have initial estimates for all states, we can use these as the next state values, v(s’), in the next iteration of our Bellman update equation. At each iteration of policy evaluation the policy, transition probabilities and rewards for moving to each state remain fixed, therefore it will only be the estimated next state values that change.
Consequently at the next iteration, with a discount factor of 1, the estimated state values for each of the possible actions for the center square become:
And for the stochastic policy, where each action is equally likely, the state value is:
Repeating this for all states gives us our second set of estimates and this time the estimated state values do take the next state into account:
If we repeat this process and run to convergence (or at least until the changes in the state values fall below some small threshold value), then we end up with the final state values shown in Figure 5, below. In this case convergence occurred after 234 iterations of Policy Evaluation.
Finally, if we now act greedily with respect to these values, choosing the action in each state that moves to the highest valued (least negative) next state, we get the following policy:
For this very simple level, this policy, formed from acting greedily with respect to the state values after one pass of policy evaluation, is the optimal policy. For any state, if Baby Robot takes the action specified by this policy and then continues to follow this optimal policy, his expected return will be higher than following actions given by any other policy.
For a finite Markov Decision Process (MDP), where we have full information about the state transitions and rewards, we can use Dynamic Programming to turn the Bellman Equations into a set of update rules. These can then be used to calculate the state values under the current policy.
For the very simple grid level we considered above, using an initial stochastic policy, we were able to find the optimal policy by running a single iteration of policy evaluation and then improving the policy by acting greedily with respect to the calculated state values.
In practice this isn’t always possible. Instead policy evaluation and policy improvement need to be run multiple times - a technique that is labelled Generalised Policy Iteration (GPI).
Under GPI, Policy Evaluation is used to calculate the state values for the current policy. Then Policy Improvement is applied, by acting greedily with respect to these values. This improves the current policy and creates the policy for the next iteration. By repeatedly applying this cycle of policy evaluation and improvement the policy is moved ever closer towards the optimal policy, as shown below:
At each step of Generalised Policy Iteration, the current policy will be evaluated and the policy improved. After each policy improvement the previously calculated state values are no longer valid and so the process begins again. Eventually the process will converge on the optimal policy, which also means we’ll have found the optimal value function.
Generalised Policy Iteration algorithms differ in how they interleave the evaluation and improvement steps. In Policy Iteration it waits for each step to complete before starting the next one. So, at each iteration, it will only update the policy after Policy Evaluation has converged.
We can test Policy Evaluation by expanding on our simple level, adding in a few more states, puddles and walls to increase the challenge. This new level is shown below:
Additionally, we’ll switch from a stochastic policy to use a deterministic one. In this policy, shown in Figure 9, a single action is specified for each state.
Since some of the actions result in looped states, and there is no path that leads to the exit, policy evaluation will never converge. Therefore, to run policy evaluation, we either need to use discounted rewards or limit the evaluation step to a fixed number of iterations.
Choosing the discount factor approach, and applying a value of 0.9, policy evaluation converges in 75 iterations. With these generated state values we can then act greedily and apply policy improvement to generate a new policy. This gives the state values and new policy shown in Figure 10 below.
After the first run of Policy Iteration, the policy is better than the initial policy and now has the rightmost states leading to the exit. However, there’s still no path all the way from the start state to the exit. So we need to run another pass of Policy Iteration. The converged state values and new, improved, policy for this are shown below:
The policy after the second step of Policy Iteration is slightly better than the last, but still hasn’t found the optimal policy. For this we need to run for a few more steps. These are shown below, along with the number of iterations required to reach convergence (the ‘Conv’ value shown on the right) on each run of Policy Evaluation. It can be seen that the optimal policy is reached after 5 steps of Policy Iteration.
In the diagram above, note how the number of iterations of policy evaluation required to reach convergence decreases after each Policy Iteration step. This is due to retaining the previous policy’s converged state values when starting the next policy evaluation. Since the new policy is based on the previous policy, it will have similar state values. Therefore, using the previous policy’s values greatly reduces the number of iterations required to run policy evaluation on the new policy.
Policy Iteration takes an initial policy, evaluates it, and then uses those values to create an improved policy. These steps of evaluation and improvement are then repeated on the newly generated policy to give an even better policy. This process continues until, eventually, we end up with the optimal policy.
As we’ve seen, Policy Iteration evaluates a policy and then uses these values to improve that policy. This process is repeated until eventually the optimal policy is reached. As a result, at each iteration prior to the optimal policy, a sub-optimal policy has to be fully evaluated. Consequently, there is potentially a lot of wasted effort when trying to find the optimal policy.
Look again at Policy Evaluation on the very simple grid level, shown in figure 13 below. Note how the blue arrows change at each iteration. These show a potential policy, formed from acting greedily with respect to the state values that are available at the current iteration.
Previously we ran policy evaluation for this level to full convergence, taking 234 iterations, and then acted greedily, with respect to the converged state values, to get the optimal policy. However, if you look at how the policy would be updated if we instead acted greedily at each iteration, you can see that the optimal policy has actually been found after only 14 iterations. We didn’t need to run evaluation to full convergence. The number of state value calculations could have been reduced dramatically and we would still have obtained the best policy.
Many GPI algorithms use this idea to go beyond simple Policy Iteration and instead improve their policy after a reduced number of Policy Evaluation steps. Value Iteration takes this idea to the extreme, effectively reducing the evaluation stage down to a single sweep of the states. Additionally, to improve things further, it combines the Policy Evaluation and Policy Improvement stages into a single update.
This can be seen in equation 3 below:
At iteration ‘k+1’ of Value Iteration, the value of state ‘s’ is given by the value of the action, in that state, that returns the maximum expected reward.
The expected value for an action is given by the sum over the transition probabilities for that action, multiplied by the reward obtained for making that transition, combined with the value of the state where you end up. Therefore the equation can be re-written as:
In a state ‘s’, we calculate the value of each action by taking the sum over the transition probabilities for that action, multiplied by the immediate reward summed with the discounted reward of the next state s′, using that state’s current value. We then choose the largest of these calculated values and use this as the value of state ‘s’ at the next iteration ‘k+1’.
If you look back at Equation 2, where the Bellman expectation equation was expressed as an update function, you’ll see that these equations are practically identical. The only difference is, in the original Policy Evaluation equation, the next state value was given by the sum over the policy’s probability of taking each action, whereas now, in the Value Iteration equation, we simply take the value of the action that returns the largest value.
The values we calculated previously, when doing Policy Evaluation for the center square of the simple grid level, are shown below:
When evaluating a stochastic policy, where each action was equally likely, we simply averaged over these 4 values to get the value of the estimated state value at the first iteration. With Value Iteration we instead take the maximum (least negative) value, which in this case is -2.0. Doing this for all states gives the first set of Value Iteration estimates shown below:
We now continue, just as we did for Policy Evaluation, to use our calculated values from the current iteration, to calculate the state values at the next iteration and run this to convergence. The state value updates at each iteration are shown below:
Once the estimated state values have stopped changing, and convergence has been reached, we can select greedily with respect to these values to get the optimal policy. In the case of this simple level, convergence occurs and the optimal policy is found after 62 iterations.
So, where Policy Iteration would evaluate a policy and then act greedily with respect to those calculated values to get the next policy, Value Iteration effectively works without a policy. Only at the end, once the optimal value function has been found, will the optimal policy be created from the state values.
For this simple grid level it takes only 62 iterations for the state values to converge and for the optimal policy to be discovered. When you compare this against the 234 iterations that are required to run Policy Evaluation to convergence it can be seen how Value Iteration is a vast improvement over simple Policy Iteration.
Additionally, since each iteration represents a sweep over all of the states, the reduction in the total number of iterations required to find the optimal policy will be of even greater benefit in more complex environments, featuring much larger state spaces.
As an example, take a look at Value Iteration applied to the expanded grid level that we previously used to test Policy Iteration:
For this level the optimal policy is found in 25 iterations, although it takes a total of 120 iterations before the state values reach convergence. If you compare this to Policy Iteration, where it took a total of 203 iterations to find the optimal state values, you can see the large improvement in performance.
The optimal policy and final, converged, state values for this level are shown in Figure 17 below. Since we didn’t use a discount factor when running Value Iteration, for each state, these values represent the average number of time steps it should take for Baby Robot to reach the exit from that state.
With Policy Evaluation we were already able to evaluate the states in a Reinforcement Learning environment. By adding Policy Improvement we can then take these values and form a new, improved, policy.
This combination of Policy Evaluation and Policy Improvement is known as Generalised Policy Iteration (GPI) and, through its application, an initial policy can be repeatedly evaluated and improved until the optimal policy is found.
In Policy Iteration, at each step, policy evaluation is run until convergence, then the policy is updated and the process repeats.
In contrast, Value Iteration only does a single iteration of policy evaluation at each step. Then, for each state, it takes the maximum action value to be the estimated state value. Once these state values have converged, to the optimal state values, then the optimal policy can be obtained. In practice this performs much better than Policy Iteration and finds the optimal state value function in much fewer steps.
In an environment where we know exactly the probabilities of moving from one state to the next, and the rewards that will be obtained for each of these transitions, we can apply Dynamic Programming to evaluate each state in the environment, and from this we can find the optimal policy.
However, these methods don’t actually learn anything. The complete model of the environment is given and so it’s not necessary to perform any exploration to form this model.
In the next part we’ll look at Monte Carlo methods, which don’t require complete knowledge of the environment and instead learn from experience.
< Part 2: Markov Decision Processes and Bellman Equations