How to Render OpenAI-Gym on Windows
By: Jian Xu
Tags: Windows, OpenAI, Docker, AI, Reinforcement Learning

How to Render OpenAI-Gym on Windows

Photo by Danielle Cerullo on Unsplash

OpenAI Gym is a great place to study and develop reinforced learning algorithms. It provides lots of interesting games (so called “environments”) that you can put your strategy to test. For example, it has simple games like balancing a vertical pole on a little cart (“CartPole-v1”), swinging up a pendulum to upright position (“Pendulum-v0”), as well as some classic video games like the Space Invader, and Pin Ball.

Image by Author, rendered from OpenAI Gym environments

However, the Gym is designed to run on Linux. Even though it can be installed on Windows using Conda or PIP, it cannot be visualized on Windows. If you run it on a headless server (such as a virtual machine on the cloud), then it needs PyVirtualDisplay, which doesn’t work on Windows either. Therefore, playing the OpenAI Gym on Windows is inconvenient.

So how can Windows users visualize the Gym?

One easy solution is to use Google Colab. It’s accessible through a web browser, and most packages are already set up properly. However, the free Colab is not without drawbacks:

Frequent session timeout and disconnection. Your session may timeout after a meal and all your data will be lost. Slow machines. Even my Dell XPS laptop runs 2x faster than the free Colab machine on many neural net models. If you already have a decent machine, why not use it? Inconvenient to mount local drives. This needs to be done every time you start a session.

After using Colab for a while, I became more and more annoyed, especially when the Colab sessions crashes more often that I’d like. Therefore, I started to consider alternative paths, but none of them seems particularly easy:

Installing a Linux/Windows dual boot is the cleanest solution. However, if you just want to do some coding from time to time, then setting up a complete operating system seems an overkill.

Installing a Linux VM should take less effort than a dual boot, but it still requires considerable amount of efforts to set up the whole virtual system, as well as dealing with potential compatibility issues, resolving libraries and package dependencies, etc.

Using a docker image should be easy, but an image from a quick search [1] somehow cannot run on my machine. Another one [2] only has “experimental” Windows support and requires to build a seemingly complicated docker image from source. As we all know, building from source on another operating system is far from straight forward.

A pure Windows solution is described in details in this article by Genevieve Hayes [3]. It uses a combination of Visual Studio, Conda, Pip, and Xming. It is not a short list of steps. I followed the steps one by one, but still I cannot build the atari-py, nor the box2d. In addition, I’m a little concern that a Windows native solution may require some source code modifications in the future.

After evaluating many options, I ended up building a minimalistic docker image myself. It’s not nearly as hard as it sounds. But if you want to skip the work of setting up environment, you can directly enjoy playing the Gym in the following three easy steps:

Download and install Docker Desktop https://www.docker.com/products/docker-desktop In a terminal, pull my docker image from the Dock Hub:

docker pull jxu305/openai_gym_docker:v1.0

3. Run the docker image from command line (and mount your favorite files if you wish)

docker run -p <local port>:8888 -it -v <local directory to mount>:<target directory> jxu305/openai_gym_docker:v1.0

or just launch from the Docker Desktop GUI

Screen capture from Docker Desktop GUI

That’s it!

For the convenience of testing, I included a small notebook in the image to test the rendering 6 example environments: Cart Pole v1, Mountain Car v0, Pendulum v0, Lunar Lander v2, Space Invaders v0, and Car Racing v0.

Image by Author, rendered from OpenAI Gym environments

In addition, I copied the Chapter 18 reinforced learning notebook from Géron, Aurélien’s book: Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow. So anyone can immediately start playing with this excellent tutorial.