Robot Goes Wild: Delta Robot Bounces Ball using Deep Reinforcement Learning
By: Rico Ruotong Jia
Tags: Reinforcement Learning, Delta Robot, Fun, Deep Learning

No one knows what the right algorithm is, but it gives us hope that if we can discover some crude approximation of whatever this algorithm is and implement it on a computer, that can help us make a lot of progress — Andrew Ng

Delta robot is a quite popular robot: it is widely used in industries, such as packaging, electronics assembly, also it can be used to draw a portrait, like Sketchy. Its simple structure makes it easy to control for handling lightweight objects in high speed motions. This nice little robot is something everyone can build in their home!

Salute in Delta Style, Photo Credit: gfycat.com

Recently, I got to see some really cool tricks done by deep reinforcement learning (DRL), such as a superhuman player playing the Atari Game. Inspired by those videos, I was wondering if we can use a delta robot for some fun tricks that is quite difficult for humans, too. Meanwhile, at Northwestern University, I was taking COMP 469, a deep learning class with Dr. Bryan Pardo and my pal, Yipeng Pan. After some discussion, we decided to make the robot bounce the ball like a trampoline. That’s how the story began.

The robot has 3 legs, some geometry analysis (roboticists also call it “inverse kinematics”) has shown that the robot top platform (aka the“end-effector”, shown in grey in the illustration below) can only travel along the x, y, z directions in space and cannot rotate. This means, we just need to control 3 joints at near the bottom base of this robot. In real life, the motors we mount at each joint has torque τ as a measurable output, so we choose the torque at each motor as the variable we control. It is quite a difficult task for humans, as we are not directly controlling the angle of each joint, and more importantly, we have only two hands but we have 3 legs to control.

Delta Robot Analysis, Photo Credit: Dr. Matt Elwin from Northwestern University

How do we go about this? After some discussions, we agreed that we needed to build 1. an OpenAI Gym Environment 2. a Deep Reinforcement Learner on PyTorch. For that, we choose to use Deep Deterministic Policy Gradient (DDPG) algorithm. Here is the original paper. After implementing the project, we also wrote a report that reflect our thoughts on the process and some challenges we faced.

OpenAI Gym On PyBullet

This is a very fun part. The physics engine I use is PyBullet 3. It’ simulates things with relatively lower CPU usage with very decent precision. My environment is ready for download, make sure you have Python 3.6 or above, you can either build from source

or Download through Pip

pip3 install gym_delta_robot_trampoline

As the first step, I built the PyBullet simulator of the model. A tutorial on how to build a parallel manipulator in PyBullet is coming soon. In short, this process goes:

Constructing a description of the model in Unified Robot Description Format (URDF) Load the URDF into PyBullet Create “constraints” to connect the top platform and all 3 legs. The main challenge in this process is URDF only supports a “serial chain” robot structure, i.e, one link can only have one parent link, so we can only connect the top platform to at most one leg at a time.

After the simulator is completed, the second step is to have it provide the information the OpenAI Gym environment needs. OpenAI has a pre-built API framework during training. The necessary information from the PyBullet Simulator is: [reward, joint positions, joint velocities, ball positions, ball velocities] (yes, this is “God View”, but in real life this can be achieved by object tracking techniques in computer vision). During each update step, the PyBullet simulator takes joint actions [τ0, τ1, τ2] as input from OpenAI Gym.

In terms of how the gym is implemented, please check out my github: https://github.com/RicoJia/gym-delta-robot-trampoline

An important note for training is: shape the reward function explicitly to induce the robot to learn the intented behavior. Oftentimes, the robot does not learn what is not reflected in the reward. To make the robot learn to bounce as many times as possible, we shaped our incremental reward as:

+10 if the ball gets over $H$ above the end-effector

+0 if the ball is above the platform but by less than a height threshold, H

-0.1 for every timstep that the robot is idle

-100 if the ball falls below the end-effector. Once the ball falls below the end-effector, the game is over.

As a side note, Gerard Maggiolino and Mahyar Abdeeteda have some awesome tutorials on how to create a OpenAI environment on PyBullet as well as how to package it for pip installation.