Reinforcement Learning: let’s teach a taxi-cab how to drive | by Valentina Alto | Towards Data Science

Reinforcement Learning is a subfield of Machine Learning whose tasks differ from ‘standard’ ways of learning. Indeed, rather than being provided with historical data and make predictions or inferences on them, you want your reinforcement algorithm to learn, from scratch, from the surrounding environment. Basically, you want it to behave as you would have done in a similar situation (if you want to learn more about the structure of RL, click here to read my former article).
In this article, I’m going to show you how to implement an RL solution using Python and its library gym-OpenAI, which you can easily install by running on your Jupyter console pip install gym. The problem I’m going to present to you is the following:
Your environment consists of a 5x5 matrix, where each cell is a position your taxi can stay at. Then, you have 4 coordinates which represent pick-up and drop-off locations, which are (0,0), (0,4), (4,0), (4,3) (for the sake of coherence with Python language, the first index is 0 rather than 1). We will refer to them as R,G,Y,B and we will index their location with, respectively, 0,1,2,3. Finally, there is one passenger which can be either picked up or dropped off, as well as being transported (hence spending time on the cab). Specifically, this passenger wants to reach point B.
Now, if we import our gym module and initialize the taxi environment, we can see that it replicates what we have been saying so far:
import gymenv = gym.make("Taxi-v2").envenv.render()
As you can see, we have our 5x5 space with our 4 locations, where the blue letter represents the current passenger’s location, while the purple one is the drop-off location. We also have our taxi/agent in that space, which is the yellow rectangle, as well as some walls, represented by the symbol ‘|’.
Now there are two elements which need our attention: states and actions.
Let’s first examine our actions. According to the module imported, the agent can act in 6 ways:
0: going down (south)
1: going up (north)
2: going right (east)
3: going left (west)
4: picking up
5: dropping off
Secondly, how many states do we have? Well, at least 25: indeed, having a 5x5 space, we know that the cab can simply occupy those cells. Furthermore, the taxi can be also in a state of picking up or dropping off the passenger (regardless of its being actually there: remember that the taxi will proceed by attempts), hence we have 4 more states. Finally, we have to compute those states where the passenger is actually picked up, dropped off (+ 4 states, since the locations where the passenger could be are 4) or simply transported (+ 1 state). So, in total, we have 5x5x4x5=500 states.
Each state of our agent, which is represented by a vector of values [row of taxi, col of taxi, passenger index, destination index], is consequently encoded with a value between 0 and 499. Namely, we can replicate the location of our previous picture like so:
state = env.encode(4, 2, 3, 2) print("State:", state)env.s = stateenv.render()
As you can see, knowing that our taxi is in position (4,2) and that passenger’s index=3 and drop-off location=2, we can derive that the encoded state is 454. For the following experiment, we will use this starting point, but before diving into that, we need to introduce one last element: the rewarding system.
The rewarding system is the main idea behind reinforcement learning: the agent is rewarded any time it acts well, otherwise it is ‘punished’ with a negative reward. In this specific case, an embedded rewarding table P is created as soon as the env is created. The logic is the following:
if the taxi correctly pick-up/drop-off the passenger, it is rewarded with +20 points
if the taxi does an illegal pick-up/drop-off, it is punished with -10 points
for each step which does not include the states above, it loses 1 point
So let’s see how it looks like for our state 454:
env.P[454]
The first thing to notice is that each entry of our P table is a dictionary with the structure {action: [(probability, nextstate, reward, done)]}
Action: it ranges from 0 to 5
Probability: in this case, is always 1
Nextstate: it is the state which occurs if that action is done
reward: the reward/penalty which is linked to that action
done: if True, it means the episode it’s over, otherwise it is not.
Let’s try to read our result: the first line is telling us that, if we go down (action 0=south), we will remain in the same position since we have a boundary, hence the reward is -1 and the episode is not over; the second line, which corresponds to the action=north, will bring our taxi towards the position 354, but the reward is always -1 and the episode is not over. The reasoning is the same for all the actions. Note that if the action is picking up or dropping off, since the taxi is not in the proper locations (R,Y,G,B), like in the last two lines (corresponding to actions 4 and 5) it receives a penalty of -10.
Now it’s time to train our algorithm. The algorithm we are going to use is called Q-learning. I’ve already provided an explanation of the idea behind in this article, hence here I won’t dive any deeper.
The procedure is explained by the following code:
import random# setting yperparameterslr = 0.1  #learning rategamma = 0.6 #discount factorepsilon = 0.1 #trade-off between exploration and exploitationfor i in range(1, 1000):  #we will see 1000 episodes    state = env.reset()  #let's reset our envepochs, penalties, reward, = 0, 0, 0    done = False        while not done:        if random.uniform(0, 1) < epsilon:            action = env.action_space.sample() # explore action space        else:            action = np.argmax(q_table[state]) # exploit learned valuesnext_state, reward, done, info = env.step(action)                 old_value = q_table[state, action]        next_max = np.max(q_table[next_state])                new_value = (1 - alpha) * old_value + lr * (reward + gamma * next_max)        q_table[state, action] = new_valueif reward == -10:            penalties += 1state = next_state        epochs += 1        
Now, imagine you have to decide which is the action which maximizes your utility (translated, which leads to the best way possible towards your passenger in position 3). Your answer will probably be north, hence action 1: indeed, it will be the fastest way to reach the location (4,3), where your passenger is located. What would say our algorithm?
np.argmax(q_table[454]) #argmax function return the position of the #maximum value among those in the vector examined
1
As you can see, the argmax function return position 1, which corresponds to action ‘north’. So, for each position, our q-table will tell us which is the action which maximizes current and future rewards.