How to Render OpenAI-Gym on Windows | by Jian Xu | Towards Data Science

OpenAI Gym is a great place to study and develop reinforced learning algorithms. It provides lots of interesting games (so called “environments”) that you can put your strategy to test. For example, it has simple games like balancing a vertical pole on a little cart (“CartPole-v1”), swinging up a pendulum to upright position (“Pendulum-v0”), as well as some classic video games like the Space Invader, and Pin Ball.
However, the Gym is designed to run on Linux. Even though it can be installed on Windows using Conda or PIP, it cannot be visualized on Windows. If you run it on a headless server (such as a virtual machine on the cloud), then it needs PyVirtualDisplay, which doesn’t work on Windows either. Therefore, playing the OpenAI Gym on Windows is inconvenient.
So how can Windows users visualize the Gym?
One easy solution is to use Google Colab. It’s accessible through a web browser, and most packages are already set up properly. However, the free Colab is not without drawbacks:
Frequent session timeout and disconnection. Your session may timeout after a meal and all your data will be lost.Slow machines. Even my Dell XPS laptop runs 2x faster than the free Colab machine on many neural net models. If you already have a decent machine, why not use it?Inconvenient to mount local drives. This needs to be done every time you start a session.
Frequent session timeout and disconnection. Your session may timeout after a meal and all your data will be lost.
Slow machines. Even my Dell XPS laptop runs 2x faster than the free Colab machine on many neural net models. If you already have a decent machine, why not use it?
Inconvenient to mount local drives. This needs to be done every time you start a session.
After using Colab for a while, I became more and more annoyed, especially when the Colab sessions crashes more often that I’d like. Therefore, I started to consider alternative paths, but none of them seems particularly easy:
Installing a Linux/Windows dual boot is the cleanest solution. However, if you just want to do some coding from time to time, then setting up a complete operating system seems an overkill.
Installing a Linux VM should take less effort than a dual boot, but it still requires considerable amount of efforts to set up the whole virtual system, as well as dealing with potential compatibility issues, resolving libraries and package dependencies, etc.
Using a docker image should be easy, but an image from a quick search [1] somehow cannot run on my machine. Another one [2] only has “experimental” Windows support and requires to build a seemingly complicated docker image from source. As we all know, building from source on another operating system is far from straight forward.
A pure Windows solution is described in details in this article by Genevieve Hayes [3]. It uses a combination of Visual Studio, Conda, Pip, and Xming. It is not a short list of steps. I followed the steps one by one, but still I cannot build the atari-py, nor the box2d. In addition, I’m a little concern that a Windows native solution may require some source code modifications in the future.
After evaluating many options, I ended up building a minimalistic docker image myself. It’s not nearly as hard as it sounds. But if you want to skip the work of setting up environment, you can directly enjoy playing the Gym in the following three easy steps:
Download and install Docker Desktop https://www.docker.com/products/docker-desktopIn a terminal, pull my docker image from the Dock Hub:
Download and install Docker Desktop https://www.docker.com/products/docker-desktop
In a terminal, pull my docker image from the Dock Hub:
docker pull jxu305/openai_gym_docker:v1.0
3. Run the docker image from command line (and mount your favorite files if you wish)
docker run -p <local port>:8888 -it -v <local directory to mount>:<target directory> jxu305/openai_gym_docker:v1.0
or just launch from the Docker Desktop GUI
That’s it!
For the convenience of testing, I included a small notebook in the image to test the rendering 6 example environments: Cart Pole v1, Mountain Car v0, Pendulum v0, Lunar Lander v2, Space Invaders v0, and Car Racing v0.
In addition, I copied the Chapter 18 reinforced learning notebook from Géron, Aurélien’s book: Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow. So anyone can immediately start playing with this excellent tutorial.
If you are interested in how to make this docker, please read on. It’s actually quite simple, all you need is a Dockerfile with less than 30 lines (GitHub link). Here I will go through the Dockerfile step-by-step.
First, I started from the Jupyter Dock Stack to avoid building from scratch. Since Géron’s machine learning book uses tensorflow, I decided to build on top of the jupyter/tensorflow-notebook image.
FROM jupyter/tensorflow-notebook
Next, install some libraries that are needed to render the environments.
USER rootRUN apt-get update -y && \     apt-get install -y xvfb && \    apt-get install -y python-opengl
Then install the OpenAI Gym, as well as the PyVirtualDisplay.
USER ${NB_USER}RUN pip install gym pyvirtualdisplay
That’s about it. With these few lines, you will be able to run and render Géron’s Chapter 18 reinforcement learning notebook, which uses the “Cart-Pole” environment.
I added a few more lines to the Dockerfile to support some environments that requires Box2D, Toy Text, and Atari. For example, the Classical Lunar Lander, and Space Invader environments.
RUN apt-get install -y cmake && \    apt-get install -y zlib1g zlib1g-dev...RUN conda install swigRUN pip install box2d-py atari_py pystan
And the final two lines are just copying some files into the image for the convenience of testing.
COPY ./example/env_render.ipynb /home/${NB_USER}/env_render.ipynbCOPY ./example/18_reinforcement_learning.ipynb /home/${NB_USER}/18_reinforcement_learning.ipynb
That’s all for the Dockerfile. Building this simple docker is easy too, just run the following in command line:
$ docker build -t <your tag>:<version> .
Congratulations! You just build your own OpenAI Gym docker! Now you can launch it with Docker Desktop and start playing with it.
Happy exercising in the gym ;-)