Practical Guide to DQN. Tensorflow.js implementation of DQN in... | by Ziad SALLOUM | Towards Data Science

“Practice what you know, and it will help to make clear what now you do not know”Rembrandt
The Deep Q-Network proposed by Mnih et al. [2015] is a jumpstart and building point for many deep reinforcement learning algorithms. However, despite its simplicity on the surface, nevertheless, it represents some challenges when implementing it, and when seeking solutions to problems.
This article will use Tensorflow.js as the implementing technology, for the obvious reasons:- The more you need to install stuff, the more unlikely you will make efforts to code the real algorithm.- Javascript is a nice and popular language and very intuitive for those who are familiar with C++, C#, Java, etc...- Sharing a demo on the web is easy and straightforward.
As said the algorithm by itself is quite easy, it can be summarized by the following image
The steps below describe how the algorithm really works:
1- Create Q-Network and Target-Network2- Fill the Experience Buffer with data using the Q-Network3- Repeat the following steps a sufficient number of times4- Get a random sample from the Experience Buffer5- Feed the sample as input to the Q-Network and Target Network 6- Use the output of the Target Network to train the Q-Network (i.e. the output of the Target Network will play the role of the labels for the Q-Network in a standard supervised learning scenario)7- Apply Exploration / Exploitation strategy (ex: Epsilon Greedy)8- If Exploration is selected then generate a random action, else If Exploitation is selected then feed the current state to the Q-Network and deduce action from the output.9- Apply action to the environment, get the reward, and the new state10- Store the old state, action, reward, and new state in the Experience buffer (also called Replay Memory)11- Every few episodes, copy the weights from Q-Network to the Target-Network
It is quite obvious to wonder, why do we need two networks?Actually, the Q learning formula is as follows:
We notice that in order for Q(S, A) to reach its final value, it must become almost equal to R’ + γ max Q(S’,a). So to make Q(S, A) move towards R’ + γ max Q(S’,a), we transform the problem into some kind of supervised learning where we estimate Q(S,A) and R’ + γ max Q(S’,a) become the ‘label’.The weights of the network used to estimate Q(S,A) will get updated via a combination of a Loss function and backpropagation.
But we don’t know the max Q(S’,a), we simply estimate it using the same neural network, however, the weights of this network are changing constantly which gives different values of max Q(S’,a) with each iteration.That makes the estimation of Q(S, A) chasing a moving target.
To solve this issue, we create an identical neural network we call it Target network, while the first network needed to compute Q(S,A) will be called Online network. The aim of the Target network is to have fixed weights for some time in order to stabilize the computation of max Q(S’,a). Then after some iterations, the Target network gets its weights updated by a simple copy of weights from the Online network that the latter had the occasion to learn.
Normalizing inputs. It might be obvious, but beginners might easily forget about this important feature. This would have big consequences on the output since some inputs might have a large range of values while others have a smaller range of values. That does not mean the latter ones are less important than the former. However computation-wise, the large values will crush the small ones. So if you see weird values for some obvious input, it is a good reflex to check if the input has been normalized.
Synchronizing the two Neural networks. DQN uses two neural networks, one that is “online” which means computing Q value at each iteration, the other network, “target”, computes the Q value but is based on a stable set of weights.The “target” network represents the expression R + γ max Q(St+1). The output of the “online” and “target” networks are used to compute the Mean Squared Error (MSE) and then update the weights of the “online” network via backpropagation.Every certain number of iterations the weights of the “online” network should be copied to the “target” network so that the “target” gets the experience learned by the “online” network.The important value here is to set the right value of iterations at which the weights are copied. Too small values, make the system unstable, too large value makes the system unable to learn. So when things are not going in the right direction, it is important to check this parameter.
Filling the experience buffer. At the beginning of the process, it is important to fill the experience buffer by letting the agent interact with the environment and store the current state, action, reward, next state, terminal state, in the experience buffer so that when the learning process generates random samples from this experience.
Sampling size. It is good practice to put the sampling size around half of the experience buffer size because having a small sampling size will make learning from experience inefficient while putting a large one (around the maximum) introduces bias in the learning that you don’t want. For example, if you have a self-driving car, and since most of the time the roads are straight, excessively sampling from straight road experience will make the AI tend to minimize the effect of turnings.
A classical implementation of the DQN is the CartPole game, where the AI agent should move the cart in a way to keep the pole near vertical.The pole must not fall beyond 30° from the vertical and the cart must not reach the edges. Winning is achieved whenever the AI agent collects 300 rewards.
You can check a real implementation of the game at https://rl-lab.com/cartpole/
The following are code snippets of the main part of the algorithm. It contains the main loop and the playStep and the training base on the batch sample.
The train() method contains the main loop of the training. It first initializes the agent and fills the replay memory buffer with experience, then it starts looping: training, then one step playing.
train(agent, maxIterations, batchSize, gamma, learningRate, syncEveryFrames) {  // init the agent  agent.reset();  // at first fill the replay memory with experience by playing  // without any training  for (let i = 0; i < agent.replayBufferSize; ++i) {    agent.playStep();  }  // create optimizer for the training    const optimizer = tf.train.adam(learningRate);  // loop until all iterations are done  let counter = maxIterations;  while (counter > 0) {    // train the agent by extracting samples from replay memory and     // use them as input for the DQN       agent.trainOnReplayBatch(batchSize, gamma, optimizer);    // after each training, play one step of the game        agent.playStep();    if (agent.frameCount % syncEveryFrames === 0) {       copyWeights(agent.targetNN, agent.onlineNN);    }    counter--;  }}
The playStep() method, makes the agent play the game for only one step. It uses an epsilon greedy policy that makes it choose random action part of the time and the action that returns max Q value the rest of the time.
// play one step of the game and store result in replay memoryplayStep() {...  let action;  let actionIndex = 0;  const state = this.game.getState();  if (Math.random() < this.epsilon) {    // Pick an action at random.    actionIndex = this.getRandomAction();    action = ALL_ACTIONS[actionIndex];  } else {  // Greedily pick an action based on online DQN output.  // use tf.tidy() to clean up after finishing     tf.tidy(() => {       const stateTensor =  getStateTensor(state);       actionIndex = this.onlineNN.predict(stateTensor)                              .argMax(-1).dataSync()[0];       action = ALL_ACTIONS[actionIndex];     });  }  // play one step of the game and get the results  const result = this.game.step(action);  // store experience in replay Memory  this.replayMemory.append([state, actionIndex, result.reward,                                 result.state, result.status]);  // if terminal state, reset the game  if (r.status != 0) {this.reset();}}
The trainOnReplayBatch() extracts a sample from the replayMemory and uses it as input into the DQN, then uses Mean Squared Error as a loss function to update the weights.
// This method is used to train the online network by using batch// samples from the memory replaytrainOnReplayBatch(batchSize, gamma, optimizer){// Get a batch of examples from the replay buffer.const batch = this.replayMemory.sample(batchSize);//define the loss functionconst lossFunction = () => tf.tidy(() => {  // example[0] is the state  // example[1] is the action  // example[2] is the reward  // example[3] is the next state  // example[4] indicates if this is a terminal state  const stateTensor = getStateTensor(batch.map(                                     example =>   example[0]));  const actionTensor = tf.tensor1d(batch.map(                                   example => example[1]), 'int32');    // compute Q value of the current state  // note that we use apply() instead of predict  // because apply() allow access to the gradient  const online = this.onlineNN.apply(stateTensor, {training: true});  const oneHot = tf.oneHot(actionTensor, NUM_ACTIONS);  const qs = online.mul(oneHot).sum(-1);    // compute the Q value of the next state.   // it is R if the next state is terminal  // R + max Q(next_state) if the next state is not terminal  const rewardTensor = tf.tensor1d(batch.map(                                         example => example[2]));  const nextStateTensor = getStateTensor(batch.map(                                         example => example[3]));  const nextMaxQTensor =  this.targetNN.predict(                                         nextStateTensor).max(-1);  const status = tf.tensor1d(batch.map(                          example => example[4])).asType('float32');  // if terminal state then status = 1 => doneMask = 0  // if not terminal then status = 0 => doneMask = 1  // this will make nextMaxQTensor.mul(doneMask) either 0 or not  const doneMask = tf.scalar(1).sub(status);  const targetQs = rewardTensor.add(                           nextMaxQTensor.mul(doneMask).mul(gamma));    // define the mean square error between Q value of current state  // and target Q value  const mse = tf.losses.meanSquaredError(targetQs, qs);  return mse;});// Calculate the gradients of the loss function with respect // to the weights of the online DQN.const grads = tf.variableGrads(lossFunction);// Use the gradients to update the online DQN's weights.optimizer.applyGradients(grads.grads);tf.dispose(grads);}
Finally here is a video in which the DQN agent plays and wins the CartPole game by reaching the reward of 300.
DQN is one of the most popular Deep Reinforcement Learning algorithms. It achieved for the first time superhuman level performance on an Atari game.As the years passed, it has been subject to many improvements, which makes it still among the best-performing DRL agents.