Deep Reinforcement Learning for Video Games Made Easy | by Andreas Holm Nielsen | Towards Data Science

In this post, we will investigate how easily we can train a Deep Q-Network (DQN) agent (Mnih et al., 2015) for Atari 2600 games using the Google reinforcement learning library Dopamine. While many RL libraries exist, this library is specifically designed with four essential features in mind:
Easy experimentation
Flexible development
Compact and reliable
Reproducible
We believe these principles makes Dopamine one of the best RL learning environment available today. Additionally, we even got the library to work on Windows, which we think is quite a feat!
In my view, the visualization of any trained RL agent is an absolute must in reinforcement learning! Therefore, we will (of course) include this for our own trained agent at the very end!
We will go through all the pieces of code required (which is minimal compared to other libraries), but you can also find all scripts needed in the following Github repo.
The general premise of deep reinforcement learning is to
“derive efficient representations of the environment from high-dimensional sensory inputs, and use these to generalize past experience to new situations.”
- Mnih et al. (2015)
As stated earlier, we will implement the DQN model by Deepmind, which only uses raw pixels and game score as input. The raw pixels are processed using convolutional neural networks similar to image classification. The primary difference lies in the objective function, which for the DQN agent is called the optimal action-value function
where rt is the maximum sum of rewards at time t discounted by γ, obtained using a behavior policy π = P(a∣s) for each observation-action pair.
There are relatively many details to Deep Q-Learning, such as Experience Replay (Lin, 1993) and an iterative update rule. Thus, we refer the reader to the original paper for an excellent walk-through of the mathematical details.
One key benefit of DQN compared to previous approaches at the time (2015) was the ability to outperform existing methods for Atari 2600 games using the same set of hyperparameters and only pixel values and game score as input, clearly a tremendous achievement.
This post does not include instructions for installing Tensorflow, but we do want to stress that you can use both the CPU and GPU versions.
Nevertheless, assuming you are using Python 3.7.x, these are the libraries you need to install (which can all be installed via pip):
tensorflow-gpu=1.15   (or tensorflow==1.15  for CPU version)cmakedopamine-rlatari-pymatplotlibpygameseabornpandas
Hyperparameter tuning for Deep Reinforcement Learning requires a significant amount of compute resources and therefore considered out of scope for this guide. Luckily, the authors of Dopamine have provided the specific hyperparameters used in Bellemare et al. (2017), which can be found in the following file. We use the contents of this “config file” as a string that we parse using the gin configuration framework. It contains all relevant training, environment, and hyperparameters needed, meaning we only need to update which game we want to run (although the hyperparameters might not work out equally well for all games).
We start by importing the required libraries
import osimport gin.tffrom dopamine.discrete_domains import run_experiment
Next, we define the root path to save our experiments
DQN_PATH = '/tmp/path/to/save/experiments/dqn'
Then, we define the game we want to run (in this instance we run the game “Pong”),
GAME = 'Pong'
And finally, we define the DQN config string:
dqn_config = """# Hyperparameters used for reporting DQN results in Bellemare et al. (2017).import dopamine.discrete_domains.atari_libimport dopamine.discrete_domains.run_experimentimport dopamine.agents.dqn.dqn_agentimport dopamine.replay_memory.circular_replay_bufferimport gin.tf.external_configurablesDQNAgent.gamma = 0.99DQNAgent.update_horizon = 1DQNAgent.min_replay_history = 50000  # agent stepsDQNAgent.update_period = 4DQNAgent.target_update_period = 10000  # agent stepsDQNAgent.epsilon_train = 0.01DQNAgent.epsilon_eval = 0.001DQNAgent.epsilon_decay_period = 1000000  # agent stepsDQNAgent.tf_device = '/gpu:0'  # use '/cpu:*' for non-GPU versionDQNAgent.optimizer = @tf.train.RMSPropOptimizer()tf.train.RMSPropOptimizer.learning_rate = 0.00025tf.train.RMSPropOptimizer.decay = 0.95tf.train.RMSPropOptimizer.momentum = 0.0tf.train.RMSPropOptimizer.epsilon = 0.00001tf.train.RMSPropOptimizer.centered = Trueatari_lib.create_atari_environment.game_name = "{}"# Deterministic ALE version used in the DQN Nature paper (Mnih et al., 2015).atari_lib.create_atari_environment.sticky_actions = Falsecreate_agent.agent_name = 'dqn'Runner.num_iterations = 200 # 200Runner.training_steps = 250000 #   250000  # agent stepsRunner.evaluation_steps = 125000 # 125000  # agent stepsRunner.max_steps_per_episode = 27000 # 27000  # agent stepsAtariPreprocessing.terminal_on_life_loss = TrueWrappedReplayBuffer.replay_capacity = 1000000WrappedReplayBuffer.batch_size = 32""".format(GAME)
And that is basically it!
Now, we just write the final code for training our agent,
Run the above (which will take a long time!), and you should see the DQN model crushing the Pong game!
We ran the experiment for roughly 22 hours on a GTX 1070 GPU. We include a visualization of the optimization results and the “live” performance of our Pong agent. We divide this into two sections:
Navigate to the tensorboard logs folder, which can be found inside the DQN_PATH that you defined earlier, and run the following:
tensorboard --logdir .
This should give you a visualization similar to this
You can see performance only gradually increases after 12 runs.
Now for the fun part!
We will use the example_vis_lib script located in the utils folder of the Dopamine library. Hence, our script for running the live demonstration looks as follows:
Run the above, and you should see the script starting to generate images for 1000 steps and then save the images into a video.mp4 file.
Here is a gif of our model:
Clearly, the agent is not perfect and does lose quite a few games. Still, it does a relatively good job!
We could probably get a close-to-perfect agent if we trained it for a few more days (or use a bigger GPU).
And there you have it!
That is essentially how little code we actually need to implement a state-of-the-art DQN model for running Atari 2600 games with a live demonstration!
Feel free to experiment with the significantly better Rainbow model (Hessel et al., 2018), which is also included in the Dopamine library, as well as other non-Atari games!
As a final demonstration, we include a small gif of an agent trained for two days for Atari Breakout using the Rainbow model:
You can see the Rainbow model is performing extremely well!
[1] Long-Ji Lin, Reinforcement learning for robots using neural networks (1993), No. CMU-CS-93–103.
[2] M. Hessel, et al., Rainbow: Combining improvements in deep reinforcement learning (2018), Thirty-Second AAAI Conference on Artificial Intelligence.
[3] P. S. Castro, S. Moitra, C. Gelada, S. Kumar, and M. G. Bellemare, Dopamine: A research framework for deep reinforcement learning (2018), arXiv preprint arXiv:1812.06110.
[4] V. Mnih, et al., (2015), Human-level control through deep reinforcement learning, Nature 518.7540 (529–533).
Originally published at https://holmdk.github.io on July 22, 2020.