Reinforcement Learning in Python with Flappy Bird | by Anthony Li | Towards Data Science

In 2014 the sleeper hit Flappy Bird took the mobile gaming world by storm. It has since been implemented in PyGame but most interestingly it lends itself well to reinforcement learning. The agent (bird) can only perform 2 actions (flap or do nothing) and is only interested in 1 environmental variable (the upcoming pipes). The simplicity of this problem makes it perfect for implementing reinforcement learning in Python from scratch.
This article gives a higher level overview of this project. The code and results can be found on GitHub here.
Q-learning is a model-free reinforcement learning algorithm which is generally used to learn the best action for an agent to take given a particular state. When the agent takes an action in a particular state it receives a reward (or penalty), and the goal of the agent is to maximise its total reward such that when taking an action it must also take the potential future reward into account. For Flappy Bird, the agent is the bird whose possible actions are to do nothing or flap, and whose current state is defined by:
The x distance to the upcoming pipes
The y distance to the upcoming bottom pipe
The current y velocity v of the bird
The distance y1 between the top and bottom upcoming pipes
The first time a new state is reached, a Q-value is initialised (0 is fine), and the agent will carry out the default action and receive a reward. If a state has been encountered before, the agent will perform the action with the highest Q-value (the action which has been rewarded the most). The Q-value is then updated and the value inserted into the respective cell in a state×action Q-table.
The Q-value is updated according to several parameters:
# Update q values in a 2d arrayQ[state, action] = Q[state, action] + alpha * (reward + gamma * np.max(Q[new_state, :]) — Q[state, action])
Alpha α is the step-size and determines to what extent the old information is forgotten in favour of new information.
Gamma γ determines the importance of future rewards. e.g. γ=0 means the agent will only consider its current reward.
Q[state, action] is the current Q-value.
The reward function was defined to penalise -1000 for a death and 0 otherwise, such that the agent’s focus is the get as high a score as possible. This ensures that the reward function has sufficient impact after each episode, vs an implementation where rewarding +1 for a score increase means that penalisation has little to no effect.
With α=0.7 and γ=0.95 the agent was initially trained for around 10,000 episodes without any exploration, and is almost able to reach a score of 1 million. The y-axis has been logged so that the rolling increase in score can be seen more clearly.
Although agent performed well after the initial training, it took exponentially longer to improve any further. This is because it takes a very long time for the agent to reach a scenario where it dies and even then it can only learn from that death once. By introducting experience replay the agent can attempt the difficult scenario multiples until it overcomes it or appears to be stuck in a resume loop (set to 100 attempts here). Upon passing the difficult scenario, upon failure it restarts from the beginning to avoid the maximum score reached from continously increasing.
There is an initial improvement in agent performance as it is able to learn from more difficult scenarios, but its performance quickly drops below its previous maximum.
This is known as catastrophic forgetting, which typically leads to an oscillation in agent performance as it unlearns and relearns the optimal actions to take. By repeatedly learning from the same scenario failure, the agent is overfitting and has forgotten the previously generalisable Q-values.
To overcome catastrophic forgetting, alpha decay is added as the agent is trained, helping it to retain the information it learnt early on whilst still learning from rarer scenarios. In addition, the number of attempts to be considered stuck in a loop is reduced to 50 and during experience replay a “replay buffer” with all the actions taken is created. The Q-table is then updated from this in a mini-batch fashion, sampling 5 of the attempts once experience replay is complete.
This agent is almost able to reach a score of 10 million. Although a drop in performance is observed as training continues past episode 10,000, it is able to recover from this initial forgetting. Whilst futher time is not spent training this agent, it would be expected that the agent performance would oscillate as it unlearns and relearns the optimal actions to take. As alpha continues to decay this should eventually enable the agent to remain stable around its maximum score.
We now try freshly trained agent, introducing the exploration rate epsilon ε that gives a chance to explore a random action until it decays from 0.1 to 0 after 10,000 episodes, and alpha decay which decays alpha from 0.7 to 0.1 after 20,000 episodes.
Due to the alpha and epsilon decay, this agent learns slower than in the initial training, but is much more stable once it has reached its optimum performance just below 1 million. This maximum score is lower than we experienced without exploration, likely due to alpha decay improving stability at the cost of the agent learning much more slowly.
From above we can see the final best performing agent was trained with experience replay and a replay buffer. Agent performance was validated over 25 runs.
The agent performs well, consistently passing 100,000 and reaching a maximum score above 5 million. It is able to pass most situations but is unable to live forever when more difficult scenarios appear, meaning it sometimes dies quite early on.
Stability: The coefficient of variation (standard deviation / mean) is 0.967. This is expected since in a random environment the agent can only be fully stable once it can overcome every scenario and never die.
Average score: The average score is 2,001,434. This is a strong score for the agent to consistently be able to reach, surpassing any human play.
Maximum score: The maximum score reached in 25 runs is 6,720,279. This is a high score close to the default maximum training value of 10 million, again surpassing any human play.
Longer training times, the best performing agent was trained for a total of 15 hours and only reached 10,674 episodes
Implement prioritized experience replay
Train an agent which never dies in the Flappy Bird environment
Tony Xu, Use reinforcement learning to train a flappy bird NEVER to die (2020), https://towardsdatascience.com/use-reinforcement-learning-to-train-a-flappy-bird-never-to-die-35b9625aaecc