Reinforcement Learning and visualisation with a simple game | by Shantanu Bhattacharyya | Towards Data Science

Games are the best way to learn the concepts of reinforcement learning (RL). But picking extremely trivial games as example doesn’t reflect the true power of the technique. On the other hand, picking a complex game with lots of graphical components makes for a cool demo but gets really hard for new practitioners to follow along.
This article will guide you through a simplistic game that uses Q-table based reinforcement learning and easy to visualise using Tkinter. The objective of the game is to make a randomly travelling green ball learn how to reach the centre of a small circle. If the ball doesn’t reach the spot within 200 steps, it loses and the game starts again.
All the code for running the RL training and its visualisation is available here
There are 3 things that needs to be defined for any RL training to happen.
Agent : It is the component of the game that will learn a policy and make decisions. For our game, the agent is the green ball.State : Think of State as a “description” of the agent at a specific point in the game. The state needs to capture all the information required to decide if the agent’s final goal (or intermediate goal) is fulfilled. This information should be necessary and sufficient. For our game, the state is captured by the 2D co-ordinates of the green ball. They are enough to tell us if the ball reached its desired final position or not.Action : Action is anything that changes the state of the agent. Usually an agent will have a finite set of possible actions it can take at any given state. The choice of best action is initially random (exploration phase) where the agent tries to figure out what are the consequences of taking the chosen action. Once the agent starts gaining experience through multiple rounds of game play, it starts developing a policy that guides its choice of action rather than random exploration. This is what we call the exploitation phase. For our game, the agent has only 8 possible directions it can choose to go to. But we have given it a choice to move each side by 10 pixels or 20 pixels. So, that makes the size of action set 16.
Agent : It is the component of the game that will learn a policy and make decisions. For our game, the agent is the green ball.
State : Think of State as a “description” of the agent at a specific point in the game. The state needs to capture all the information required to decide if the agent’s final goal (or intermediate goal) is fulfilled. This information should be necessary and sufficient. For our game, the state is captured by the 2D co-ordinates of the green ball. They are enough to tell us if the ball reached its desired final position or not.
Action : Action is anything that changes the state of the agent. Usually an agent will have a finite set of possible actions it can take at any given state. The choice of best action is initially random (exploration phase) where the agent tries to figure out what are the consequences of taking the chosen action. Once the agent starts gaining experience through multiple rounds of game play, it starts developing a policy that guides its choice of action rather than random exploration. This is what we call the exploitation phase. For our game, the agent has only 8 possible directions it can choose to go to. But we have given it a choice to move each side by 10 pixels or 20 pixels. So, that makes the size of action set 16.
Let’s examine the code for setting up the state space (set of all possible states) and action space (set of all possible actions) for our game.
mov_list = [-20,-10,10,20]#Set up Q tablestate_space = {}temp_list = []for n in range(0,500,10): for m in range(0,360,10):  temp_list.append((n,m))for k in range(1800): state_space[k] = temp_list[k]action_space = {}temp_list = []for n in mov_list: for m in mov_list:  temp_list.append((m,n))for k in range(16): action_space[k] = temp_list[k]
This code assumes a game window of 500px x 360px. So, our state space should have have a size of 18000 (every x,y pixel the agent can reach). But we have defined possible movements (mov_list)only in multiples of 10. So, that shrinks our state space 10 fold to 1800. The action space is similarly composed of all possible movements (x and y combinations) giving a set of 16.
OK, so now we have defined the agent, state and action for our game. But the question is how will the agent know whether the action it is taking is beneficial or not ? This is where the reward function kicks in. The rewards (or penalty) can be provided after each action or at the end of the game. It is generally a good idea to keep the reward system simple. For example, in our game there the ball has a maximum of 200 steps to reach the circle. There are no rewards for each step. If the ball finds the circle before the 200 steps, then a reward of 100 is given.
So, how is this reward used to create a policy ? There are multiple ways to do this but we have used Q-tables which are one of the easiest tools for policy updates. A Q-table is simply a matrix of state space x action space.
q_table = np.zeros([len(state_space),len(action_space)])
For our game, it is a matrix of 1800 x 16 initialised to 0 for every cell. If any action results in a reward, we update the Q table for the corresponding state, action pair.
old_q_value = q_table[state, action]  next_state = list(state_space.keys())[list(state_space.values()).index((pos_x1, pos_y1))]  next_max = np.max(q_table[next_state])  q_target = reward + gamma * next_max  q_delta = q_target - old_q_value  q_table[state, action] = old_q_value + alpha * q_delta
I have introduced two new terms here. ‘alpha’ sets the learning rate for the algorithm and ‘gamma’ is a discount factor for future rewards. Higher value of gamma places higher importance to possible future rewards. We have used an alpha of 0.9 and gamma of 1.
This leaves us with one last thing. Setting the exploration vs exploitation ratio. This is set by a term epsilon. If a random number (0 to 1) is below epsilon, then the agent is free to explore and ignore the current policy as per the Q table.
if random.random() < epsilon:    x1 = random.choice(mov_list)    y1 = random.choice(mov_list)    action = list(action_space.keys())[list(action_space.values()).index((x1, y1))]   else:    action = np.argmax(q_table[state])    x1, y1 = action_space[action]
The whole script is available here
The best way to visualise how the reinforcement learning is performing is to track the movement of the ball through some GUI. I have kept the RL code separated from the visualisation code so that the RL runs extremely fast. You can run the RL for certain number of games (episodes parameter in the code) and then run the GUI to see how the ball is behaving. The GUI is Tkinter based and should work easily in any platform. The main thing to remember is that epsilon is set to 0 for the RL runs. So it fully explores over all games during RL training (and builds the policy). Feel free to set it higher and observe the results on the GUI.
If you want to visualise exploration followed by progressive exploitation, check out this script in the same repo. Be careful not to set too few episodes for exploration, otherwise the ball will not learn much and start exploiting a poor policy.
That’s it ! Play around with parameters and once you get comfortable, start creating more complicated games. You will soon realise that Q table updates get extremely slow once the state space gets too big. That’s where Deep Q networks come in. A topic to be covered another day.