Reinforcement Learning — Cliff Walking Implementation | by Jeremy Zhang | Towards Data Science

The essence of reinforcement learning is the way the agent iteratively updates its estimation of state, action pairs by trials(if you are not familiar with value iteration, please check my previous example). In previous posts, I have been repetitively talking about Q-learning and how the agent updates its Q-value based on this method. In fact, besides the update method defined in Q-learning, there are more other ways of updating estimations of state, action pairs. In this post, we will together explore another method called SARSA, compare this method with Q-learning and see how the difference in update methods affects an agent’s behaviour.
Let’s first talk of temporal difference, which is the core of an updating method. We know that at each iteration or episode, an agent explores the environment by taking action following a policy(say ε-greedy), and based on its latest observation, which is summarised as a value of state-action, it updates its current estimates by tweaking the current estimation a little bit towards the latest observation, and the difference between the values of latest observation and last is called temporal difference. And it is from this temporal difference that our agent learns and updates itself.
The definition of temporal difference distinguishes methods from each other. In order to give you a more concrete sense, let’s directly dive into the algorithm definition and check the difference.
It is clear that the only difference lies in updating the Q function. In SARSA(by the way, the name SARSA comes explicitly from the process of the agent, which state, action, reward, state, action ...), the temporal difference is defined as :
[R + Q(S', A') - Q(S, A)]
where the observed Q value of next state, action pair contributes directly to the update of current state. SARSA is also called on-policy, because the update process is consistent with the current policy.
However the temporal difference defined in Q-learning is:
[R + max_a(Q(S', a)) - Q(S, A)]
where the observed Q value of next state, action pair may not directly contribute to the update of current state. The Q-learning always uses the max value of next state, in which case the state, action being taken to update the Q value may not be consistent with the current policy, thus it is called off-policy method.
The difference, in fact, could result in different behaviours of an agent. The gut feeling is that Q-learning(off-policy) is more optimistic in value estimation, where it always assume the best action be taken in the process, which, could result in bolder actions of the agent. Whereas SARSA(off-policy) is more conservative in value estimation, which result in saver actions of the agent.
To clearly demonstrate this point, let’s get into an example, cliff walking, which is drawn from the reinforcement learning an introduction.
This is a standard un-discounted, episodic task, with start and goal states, and the usual actions causing movement up, down, right, and left. Reward is -1 on all transitions except those into the region marked Cliff. Stepping into this region incurs a reward of optimal path -100 and sends the agent instantly back to the start.
This is a typical 2 dimensional board game, so the board settings are mostly same as the example I described here. In the following sections, I will mainly emphasise on the implementation of SARSA and Q-learning, and the comparison of resulted agent’s behaviours between these two methods.
In a nut shell, we will have a Cliff class which represents the board that is able to:
Keep track of an agent’s current positionGiven an action, decide the agent’s next position and judge whether it is the end of gameGive feedback as reward
Keep track of an agent’s current position
Given an action, decide the agent’s next position and judge whether it is the end of game
Give feedback as reward
These are the major function inside class Cliff. The nxtPosition function takes in an action and returns the agent’s next position on the board, and if the agent bashes its head into the wall(reaches the border), it remains at the same position. The giveReward function gives reward -1 to all states except the cliff area, where results in reward -100.
Let’s have a look at the board we implemented.
The agent starts at the left end of the board with a sign S, and the only way to end the game is to reach the right end of the board with a sign G. And * represents the cliff area.
In terms of game playing, we will have an Agent class representing our agent, and inside the agent class, there is a function decides the agent’s action taking, which is again the ε-greedy policy. (Check out full implementation here)
The key difference lies in the play function:
In each episode(each round of the game), we keep track of our agent’s action, state and reward in the list self.states . And at the end of the game, we update the Q function(the self.state_actions) in reversed fashion — if the method is SARSA(on-policy), the newly updated reward (which essentially is Q(S', A')) will be directly applied to next update, and if the method is Q-learning, there is one more step,
reward = np.max(list(self.state_actions[pos].values()))
which takes the maximum value of all actions in that position to the next round of update.The maximum operation shapes the agent behaviour and enables it to take more adventurous actions.
I ran both methods with exploration rate 0.1 for 500 rounds and took the final (state, action) they learned.
The result I learnt is slightly different from the optimal result in the book, but it is clear enough to see the difference between the two.
The conclusion part I will take a reference from Sutton’s book, which perfectly summarises the difference between the two methods:
Q-learning learns values for the optimal policy, that which travels right along the edge of the cliff. Unfortunately, this results in its occasionally falling off the cliff because of the “epsilon-greedy” action selection. SARSA, on the other hand, takes the action selection into account and learns the longer but safer path through the upper part of the grid. Although Q-learning actually learns the values of the optimal policy, its online performance is worse than that of SARSA, which learns the roundabout policy. Of course, if ε were gradually reduced, then both methods would asymptotically converge to the optimal policy.
And finally, please check out the full code here. You are welcomed to contribute, and if you have any questions or suggestions, please raise comment below!