Deep Q reinforcement learning (DQN) | Towards Data Science

Q learning is a method that has already existed for a long time in the reinforcement learning community. However, huge progress in this field was achieved recently by using Neural networks in combination with Q learning. This was the birth of so-called Deep Q learning. The full potential of this method was seen in 2013 when google presented to the world their DQN agent playing atari breakout. For me, this was the first contact with the field and I immediately got interested in it.
Other reinforcement learning methods are policy gradient methods and actor critic methods. Actor critic methods are a hybrid between Q Learning and Policy gradient methods.
In reinforcement learning, we have an environment, in which an agent is doing actions. The environment then returns a reward and a new state. The reward, the agent receives also depends on the state he is actually in. So he/she should learn to adapt the actions to the actual state.
The environment we are going to use is the CartPole-v0 environment, which is authored by OpenAI. We can easily use it in python. It is a pretty simple environment and also very simple to solve for most deep reinforcement learning algorithms. Therefore I use it every time I write a new Reinforcement learning script to test if I implemented everything correctly.
We start by importing and testing openai gym. If you haven’t installed it yet, you can install it by typing:
pip3 install gym
We then write a script to test our gym environment. We let the agent perform random actions, just to see, that everything works.
The goal of the agent is to balance the stick for as long as possible. It fails and the episode get’s terminated, if the stick diverges 15 degrees from being upright in both directions.
The actions the agent can take to achieve its goal are:
move left (0)
move right (1)
Reward is given in the following way:
+1 for every step, that the stick is upright
The state/observation is a list of 4 values:
Consider the following scenario. Our chicken is hungry. It has two actions it can perform. It can either eat or write. Since it is hungry, the better choice would be to eat and this indeed would give it more reward. But reward is given after an action is performed, so how is it supposed to know. In this scenario, the action value(or Q value) can help us. This Q value depends on the state and and a possible action. Q(s, a) then returns us the value of performing an action, given the state. So the chicken should perform the action, that has the highest q value!
The Q value is defined by the following equation:
This equation simply means, that for a given state, the value of an action is the immediate reward after the action was performed plus the (discounted) highest Q value of the next state. Gamma (γ) is actually the discount factor and it accounts for the uncertainty in future rewards. γ is usually above 0.9 and smaller than 1. A low gamma leads to the agent being more focused on immediate reward, while a high gamma leads to the agent being more focused on high future rewards.
We have seen the exact definition of the Q value and I also told you, that we are going to use a neural network to approximate it.
So we want to bring the approximated Q value of the network as close as possible to the mathematical definition. To achieve this goal, we use the Neural networks ability to minimize a loss function.
We call the Q value of the actual time step “prediction” and the term including the Q value of the next state target.
The loss function we used is simply the mean squared error loss.
With temporal difference learning, we can train our agent at every time step.
If we would let our agent greedily exploit the strategy, which it thinks has the highest Q value, it is very likely, that there exists a better strategy. The agent will never be able to explore this strategy and therefore never see the high reward associated with it. So in the beginning, we let our do random actions and over time decrease it’s probability of taking random actions. This process is called Epsiolon (ε) greedy action selection.
Deep Q learning does not especially well unless we enhance it by adding experience replay. For this, we build a memory and store all states, actions and rewards in it. When training, we retrieve a random batch from our memory and train on it. This means, that the agent will most likely not be trained on the last time step it took.
The following code implements a ReplayBuffer class, which is responsible for storing and giving out random batches of memory. For each timestep we store: (state, action, reward, next_state, done). done is a terminal flag and is 0 during an episode and 1 when the episode ends.
Instead of presenting pseudo code, I decided to present you the python code of the main file, because it is easier to understand, if you know python.
We can adjust many hyperparameter, which we pass into the constructor of the agent:
agent = DQAgent(learning_rate=0.00025, gamma=0.90, batch_size=32,                      state_len=len(env.reset()),                      n_actions = env.action_space.n,                     mem_size = 1000000,                     min_memory_for_training=1000,                      epsilon=1,epsilon_dec=0.99,                     epsilon_min = 0.02)
Learning rate: learning rate of the neural network. A high value leads to the agent learning fast, but can also lead to sub-optimal behavior in complex environments.
Gamma: γ is usually above 0.9 and smaller than 1. A low gamma leads to the agent being more focused on immediate reward, while a high gamma leads to the agent being more focused on high future rewards.
Batch size: The number of timesteps given to the neural network to perform minibatch gradient descent. High number of batches stabilizes learning, but can also lead to stagnation.
Min memory for training: Number of steps, before the learning process begins. A decent number prevents the network from overfitting to the first few training samples.
Epsilon: The start value of epsilon. 1 means agent perform only random actions (exploration) in the beginning and 0 means it would start performing no random actions(exploitation).
Epsilon dec: The factor by which epsilon is multiplied after each time step to decrease it. It should be bigger than 0.99 and smaller than 1.
Epsilon min: The minimal value for epsilon. Epsilon will not get decreased if it would fall below this value. It is very important to have a little exploration at any time, so that the agent can continue to learn.
And now the whole code of our agent:
Let’s discuss this code line:
target = rewards_batch + torch.mul(self.gamma* self.q(new_states_batch).max(axis = 1).values, (1 - dones_batch))
It calculates the TD target. Lets’s assume our batch size is 5.
Then the calculation could be looking as follows:
We do the max operation, because we want to look at the action with the highest Q value.
And Let’s also look at the following code line:
prediction = self.q.forward(states_batch).gather(1,actions_batch.unsqueeze(1)).squeeze(1)
It calculates the prediction. With a batch size of 5, this could look the following way:
We select the q values from the actions, the agent actually took.
Now it’s time to actually test our algorithm. We test it with our CartPole-v0 environment.
After 100 episodes, the agent has managed to beat the environment and almost always achieves the highest score.
To improve our deep Q learning algorithm even further, we can choose to have separate prediction and target values. This is also called double Q learning.
The actions will be predicted with the prediction network. Backpropagation only happens in the prediction network. The parameters of the prediction network get copied every few iterations to the target network. How long the parameters of the target network remain “frozen” is also a hyperparameter.
We freeze the target network, because then the prediction and the target are less correlated. This helps in the learning process
Now we have a little fun and test our algorithm with the openai environment “LunarLander-v2”. You can find a list of the hyperparamters which I have used below.
Learning rate: 0.001
Gamma: 0.99
Batch size: 64
Min memory for training: 1000000
Epsilon: 1
Epsilon dec: 0.995
Epsilon min: 0.02
Frozen_iterations: 6
The following GIF was taken at episode 729. The agent does very well at landing that lander.
And the learning curve looks like this.
Note that it was smoothed by a 20 episode moving average.
The main file:
Code for the agent:
towardsdatascience.com
towardsdatascience.com
towardsdatascience.com
Linkedinhttps://www.linkedin.com/in/vincent-m%C3%BCller-6b3542214/Facebookhttps://www.facebook.com/profile.php?id=100072095823739Twitterhttps://twitter.com/Vincent02770108Mediumhttps://medium.com/@Vincent.MuellerBecome medium member and support me (part of your membership fees go directly to me)https://medium.com/@Vincent.Mueller/membership