Dynamic Programming in RL. Towards Training Better Reinforcement... | by Rohan Jagtap | Towards Data Science

Dynamic Programming is a mathematical optimization approach typically used to improvise recursive algorithms. It basically involves simplifying a large problem into smaller sub-problems. There are two properties that a problem must exhibit to be solved using dynamic programming:
Overlapping SubproblemsOptimal Substructure
Overlapping Subproblems
Optimal Substructure
We’ll be discussing ‘Planning in RL’ using dynamic programming. Planning mainly requires the complete environment’s knowledge (usually an MDP) or a model of the environment in advance. And using this knowledge, we can solve for the optimal policy.
In my previous article on Reinforcement Learning, I have covered the formulation of RL problems as a Markov Decision Process (MDP). You can consider giving it a read if you’re unaware of what an MDP is.
In this article, we will take the example of a Grid World Game and try to win it using ‘planning’. But first things first, let’s talk about how are we going to approach this problem:
Prediction: In this step, we evaluate our policy, i.e., we evaluate the value function for all the states.Control: In this step, we use the computed value function to improve our policy.
Prediction: In this step, we evaluate our policy, i.e., we evaluate the value function for all the states.
Control: In this step, we use the computed value function to improve our policy.
We use the above steps iteratively to arrive at the optimal policy π∗. I’ll be covering 2 methods for planning:
Policy Iteration andValue Iteration
Policy Iteration and
Value Iteration
In this problem, we are given a grid (4 x 4 in this case). The goal is to reach either the top-left or the bottom-right square (gray colored) from any other square on the grid, with maximum reward.
You can jump one square in either of the North, South, East, or West directions from any given square.
Each jump (in any direction) earns a reward of -1 except from the gray squares where the reward is 0.
Jumping off the grid earns you -1, but you stay in the same square.
Once you reach the goal, the game is over, and you cannot jump anywhere (i.e., even on jumping, you stay in the same square with a 0 reward).
To approach this problem, we consider our grid to be an MDP, where each square is a state. The state transition reward for all the states is -1 except for the terminal states where the reward is 0. Our goal is to find an optimal policy for this MDP.
The algorithm is pretty straight forward:
Given a policy π,
Evaluate the policyImprove the policy
Evaluate the policy
Improve the policy
To improve a policy, we first need to evaluate it. So in this section, we will use Bellman’s Expectation Equation iteratively to evaluate the value function for all the states in an MDP for a given policy.
In our grid-world problem, we will start with a random policy initially, i.e., π(North|.) = π(South|.) = π(East|.) = π(West|.) = 0.25 (. means any state). To evaluate the policy:
At each iteration (k + 1):    For all states s ∈ S    Update v_(k + 1)(s) using Bellman's Equation as follows:
Initially, we assume that all the states yield nothing, i.e., we initialize the state value for all states with 0. Let’s start:
Now, we will iteratively update these values with a one-state lookup for each state. By one state lookup, I mean just considering the state value of the immediate next state with a discount (γ) of 1.
For understanding, we will calculate the state value for just one state (circled in red).
With respect to the Bellman’s equation, we have:
Note that v_k(s’) is 0 for all the north, south, east and west states. In case of north, as mentioned earlier, going off the grid results in the same state with a reward of -1. Therefore:v_(k+1)(s) = 0.25(-1 + (1)(1)(0)) + 0.25(-1 + (1)(1)(0)) + 0.25(-1 + (1)(1)(0)) + 0.25(-1 + (1)(1)(0))v_(k+1)(s) = 0.25(-1 + -1 + -1 + -1) = -1.0
We repeat this for all the states in the MDP:
Based on the values we calculated in the previous iteration we have:
v_k(s’) for north, south and east states is -1 and for the west state, it is 0. Therefore:v_(k+1)(s) = 0.25(-1 + (1)(1)(-1)) + 0.25(-1 + (1)(1)(-1)) + 0.25(-1 + (1)(1)(-1)) + 0.25(-1 + (1)(1)(0))v_(k+1)(s) = 0.25(-2 + -2 + -2 + -1) = -1.75
Note that the values in the figures are rounded of to one decimal place:
Similarly we compute for k = 3:
v_k(s’) for south and east states is -2.0, for north it is -1.7 and for the west state, it is 0. Therefore:v_(k+1)(s) = 0.25(-1 + (1)(1)(-1.7)) + 0.25(-1 + (1)(1)(-2)) + 0.25(-1 + (1)(1)(-2)) + 0.25(-1 + (1)(1)(0))v_(k+1)(s) = 0.25(-2.7 + -3 + -3 + -1) = -2.425
If we keep iterating, we will reach to a point where we can barely see any changes in the state value function. This indicates that we have converged to the true value function v_π:
In this step, we simply pick an action greedily based on our true value function:
So basically, we assign probabilities to the actions based on the max value function values of the adjacent states. Therefore, based on v_π, we have:
Turns out, this is the optimal policy for our grid-world problem! However, in most cases, we have to follow the evaluation-improvement iterative pattern and stop once the policy stops improving.
In hindsight, if you notice, we could have obtained the above optimal policy even at k = 3 in the policy evaluation.
So why go up to k = ∞? Why not update the policy after every iteration (i.e., at k = 1)? This is exactly what’s done in Value Iteration.
From the above definition, we can see that this technique uses recursion for each state accessible from s. It says that if we know the solution to subproblems v∗(s′) (s′ are all the states reachable from s), then the solution v∗(s) can be found out with a simple one state lookahead using:
Since we are assuming the optimal value for the future states, we will use the Bellman’s Optimality Equation (as opposed to the Bellman’s Expectation Equation used in Policy Iteration).
The idea is to go backwards (i.e., start from the final state and then trace the path back). This will ensure the optimal value for the future states, as we are calculating them first.
Just like policy iteration, we initially assume that all of the states yield nothing, i.e., 0:
Now, we will take the optimality equation and calculate the state values iteratively:
Note that v∗(s’) is 0 for all the north, south, east, and west states. Also, γ is 1. Therefore:v∗(s) = max(-1 + (1)(1)(0), -1 + (1)(1)(0), -1 + (1)(1)(0), -1 + (1)(1)(0))v∗(s) = max(-1, -1, -1, -1) = -1
We do this for all the states in the MDP:
Since we are going backwards, we now have the optimal values for the states adjacent to both goals. In the next step, we calculate the values for all the states except for these states (I’ll explain why in a while). So we will obtain the optimal values for the states adjacent to these states, and so on... Therefore:
If you notice, now we have the optimal policy at k = 3.
Let’s trace the path for the top left goal state. In the first iteration, we calculate the state values as we did for policy iteration.
As said earlier, value iteration is equivalent to updating the policy at k=1 in policy iteration.
Hence, after this step, our updated policy will have probability 1 to go towards the goal for the following two states:
Similarly, at k = 2, we will obtain the optimal value for the next states, and clearly, the optimal policy shows:
And finally, at k = 3:
we can see that two of these states are also influenced by the other goal (bottom right square) and hence the outgoing arrows.
Notice that the state values in Value Iteration are more of a ‘Shortest Path’ from any state to the goal, contrary to Policy Iteration where the values are the ‘Expected Values’ for all the states.
In this article, we discussed 2 techniques for solving for the optimal policy using the dynamic programming paradigm. However, in these examples, we did not discuss the formulation of the reward. The whole MDP was handed to us, and we just did the ‘planning.’ Moreover, we already knew the final states; hence we could solve backward in value iteration. In many problems, we don’t even know if a goal exists. There are other additions to these techniques that we use in such cases. We will discuss these techniques in the future.
towardsdatascience.com
David Silver’s Lecture on MDPs: https://www.youtube.com/watch?v=lfHX2hHRMVQ&list=PLqYmG7hTraZDM-OYHWgPebj2MfCFzFObQ&index=2
Sutton and Barto RL Book: https://web.stanford.edu/class/psych209/Readings/SuttonBartoIPRLBook2ndEd.pdf