Spinning Up in Deep Reinforcement Learning (With PyBullet)
By: Rive Sunder
Tags: Reinforcement Learning, Open Source, Simulation, Physics

If you want to skip the preamble and get started right away, jump down to Setting Up in Deep RL with PyBullet. You can always come back to read the intro while your experiment is running. Also, friends fix fading text here.

Perhaps you’ve decided to study deep reinforcement learning. Maybe you’re getting a jump on a New Year’s resolution, perhaps it’s something you think might help your career, or maybe you just can’t find a human that wants to play Go with you. Whatever your reasons, If you haven’t found it already, I’d like to point you toward a useful resource built by Joshua Achiam: Spinning Up in Deep Reinforcement Learning.

The hardest part of using PyBullet for reinforcement learning is getting out of bed.

Spinning Up is great for getting started with running experiments, with implementations and descriptions of several of the most well-known RL algorithms. It also includes several exercises, a bibliography, and an essay of tips and tricks for conducting good research once you get started. For the most part, the project exudes an open-ness and desire to create an environment conducive to the success of others which I find appealing, or at least that’s a general impression I get from this type of project. There’s one small problem, and it’s one that is increasingly shared by projects originating at OpenAI: the project is built to depend on MuJoCo for continuous control robotics.

MuJoCo is a phyisics simulator that was originally developed with public funding from the NSF and NIH, but is now available to use with licenses ranging from $500 US for personal use with no commercial application, to as mush as $2,000 US per person for commercial projects. There is also a 30-day trial and a free student version available, the latter of which is so limited as to be nearly useless (you can’t do research, and the license is invalid for classwork where more than 50% of the class uses MuJoCo). There is plenty of worthwhile criticism directed toward OpenAI for their insistence on pushing MuJoCo in their projects. But this isn’t going to be a pessimistic post about propping up barriers to entry in reinforcement learning and artificial intelligence research; this post is going to show you how you can use a permissively licensed physics simulator for all the canonical robotics tasks often reported in RL literature, free of charge.

PyBullet provides python bindings to the Bullet physics simulator. Released under a permissive zlib license, it’s widely used in games and research. Luckily for us, it also comes with a suite of reinforcement learning environments, including the RL robotics tasks you may have seen before. For a full list of RL environments in PyBullet, check out the Quick Start Guide.

There’s no need to re-hash a beginner’s intro to RL here. For that, head over to Spinning Up. You’ll probably also want to check out a few other of my favorite RL resources at some point: the Berkeley/OpenAI Deep RL Bootcamp from 2017 and Sergey Levine’s deep RL courses from Berkeley. If you have your own favorites, feel free to mention them in the comments.

Without further ado, let’s get started with PyBullet and Spinning Up. It’s actually going to be really easy — if you had skipped reading this introductory fluff you would already have an experiment running ;)