Neural Network Plays Flappy Bird. Currently, I am an IT student in... | by Woojae Kim | Towards Data Science

Currently, I am an IT student in college. This semester, I had a really interesting course which I choose my own topic to study, and create my own project. So I decided to learn and work on something very interesting and unique. While I was exploring various topic, I found a video tutorial about Neural Network, and I was really interested in learning this topic.
After watching the tutorial video, I come up with an idea to implement a neural network program that learns how to play Flappy Bird game. Finally, this is my result.
When the program runs first time, it does nothing because it has no idea about the game. However, a few hours of training, it learns how to play the game.
In this posting, I want to give a basic information about Neural network and how I implemented my program.
First of all, I want to talk about what neural network is. In programming, artificial neural network is computational model/algorithm for machine learning that is inspired by structure and functional aspects of biological neural networks.
Every neural network has one input layer, one output layer and one or more hidden layers. Each circle represents a neuron and a neuron has a connection to every neuron in the next layer.
Each connection has weight value and each neuron has a bias value. For example, below diagram shows what’s happening in the neural network.
When neural network calculate the output, it does a lot of math with weight and bias. You can easily expect that if I change one of the weight or bias value, the final output will also be changed. In other words, training neural network means finding and adjusting weight and bias values that gives the best output that we want.
Synaptic.js is a great JavaScript neural network library for node.js and browser. If you have node.js installed, you can install synaptic.js with using following command:
npm install synaptic --save
Now, I want to provide very brief example solving problem using Synaptic.js library.
If you take a look at table on the left, it has two input and one output. If the first input is greater than the second input, the output is 1. Otherwise the output is 0.
The first five records will be our test data and we will see what is the output for the last two records.
Following code shows how I implemented using synaptic.js to solve the problem.
var synaptic = require('synaptic');var myNetwork = new synaptic.Architect.Perceptron(2, 3, 1);var learningRate = .2;//trainingfor (var i = 0; i < 20000; i++){    //test data 1    myNetwork.activate([2,1]);    myNetwork.propagate(learningRate, [1]);    //test data 2    myNetwork.activate([1,5]);    myNetwork.propagate(learningRate, [0]);    //test data 3    myNetwork.activate([4,2]);    myNetwork.propagate(learningRate, [1]);    //test data 4    myNetwork.activate([5,2]);    myNetwork.propagate(learningRate, [1]);    //test data 5    myNetwork.activate([3,5]);    myNetwork.propagate(learningRate, [0]);}console.log(myNetwork.activate([2,5])); //[ 0.0000728000640238746 ]console.log(myNetwork.activate([3,1])); //[ 0.9999465030073619 ]
var myNetwork = new synaptic.Architect.Perceptron(2, 3, 1);
Above code creates neural network with 2 input neurons, 3 hidden neurons and 1 output neuron.
myNetwork.activate([2,1]);
activate() method returns output from the given input (in above code, 2 and 1). The output value is between 0 to 1 float value.
myNetwork.propagate(learningRate, [1]);
After call the activate function, we call propagate() function to train the network with learning rate and target output value. This is done by backpropagation.
When I implement the flappy bird program, I’ve come up with 5 inputs and 1 output.
The inputs are:
Bird Position
Velocity
Distance between next pipe
Upper pipe height
Lower pipe height
Each time the game is rendered, neural network gets the input and activate it. If the output is greater than 0.55, the bird will jumps.
However, I cannot use propagate() method because there is no test data and its target value. In this case, we can use genetic algorithm to train our neural network.
Assume that I have 8 birds in one generation and each bird has 6 neurons. (In my actual implementation, I have 12 birds in a generation and 18 neurons for each birds).
After one generation (12 games), I pick the best 4 birds and apply crossover and mutation. Repeat the same process until we get good results.
For crossover function, the bias value will be crossover-ed.
Learn.crossOverDataKey(netA.neurons, netB.neurons, 'bias');
Cut the random location of the neuron, and crossover it.
Learn.crossOverDataKey = function (a, b, key) {    var cutLocation = Math.round(a.length * Math.random());    var tmp;    for (var k = cutLocation; k < a.length; k++) {        // Swap        tmp = a[k][key];        a[k][key] = b[k][key];        b[k][key] = tmp;    }}
For mutation function, bias and weight value will be mutated.
Learn.mutateDataKeys(net.neurons, 'bias', Learn.mutationProb);Learn.mutateDataKeys(net.connections, 'weight', Learn.mutationProb);
Connection and neurons are randomly selected and mutated to random value. In my implementation, I have mutation rate 20%.
Learn.mutateDataKeys = function (a, key, mutationRate){    for (var k = 0; k < a.length; k++) {        // Should mutate?        if (Math.random() > mutationRate) {            continue;        }        a[k][key] += a[k][key] * (Math.random() - 0.5) * 3 + (Math.random() - 0.5);    }}
You can see my full code here.
Also, here are links to resources that I used:
synaptic.js: https://github.com/cazala/synapticflappybird web version: https://github.com/nebez/floppybirdcode/algorithms that I referenced: https://github.com/ivanseidel/IAMDinosaur