Accelerated Robot Training through Simulation
with ROS and Gazebo

Risto Kojcev, Alejandro Hernández Cordero, Asier Bilbao Calvo, Irati Zamalloa Ugarte,
Yue Leire Erro Nuin, Víctor Mayoral Vilches

AI Team @ Erle Robotics
We investigate the state of the art deep-learning methods,
and apply them to modular robots.

Source: Erle Robotics

AI Infrastructure: Algorithms

powered by HRIM
powered by AI

Two unlinked worlds

Sadeghi, Fereshteh, et al. CVPR 2018

Sadeghi, Fereshteh, et al. CVPR 2018

gym_gazebo

Two unlinked worlds

GYM_GAZEBO: a Link between AI and Robotic Methods
environment API

Robot toolbox
and libraries

RL Algorithm

Simulator
GYM_GAZEBO is Open Source. Try it!
https://github.com/erlerobot/gym-gazebo

Github stats
WatchingStars42395

GYM_GAZEBO: Community Projects

Nicola Ristè
https://youtu.be/h84IkUCze44

Institut de Robòtica i Informàtica

An “end to end” solution for modular robots

Reinforcement Learning requires long training time…

…this is me waiting for results !

ROBOT_GYM: accelerating training

ROBOT_GYM: workflow
OrchestrateWorkers

Initialize global
policy ( πθ )Worker 1

Worker 2

Worker 3

generates
rollouts

generates
rollouts

generates
rollouts

Merge policy (πθ )Update policy( πθ )…

ROBOT_GYM: workflow
Orchestrate
Workers

Initialize global
policy ( πθ )Worker 1

Worker 2

Worker 3

generates
rollouts

generates
rollouts

generates
rollouts

Merge policy (πθ )Kubernetes Cloud Network
Update policy( πθ )

replica
container
container…

VPN
bridge server

Local
workers…

InternetcontainerhubROBOT_GYM: workflow
Orchestrate
Workers

Initialize global
policy ( πθ )Update policy( πθ )container

Workerenvironment API

Worker 1

Worker 2

Worker 3

generates
rollouts

generates
rollouts

generates
rollouts

Merge policy (πθ )RL Algorithm

Robot toolbox
and libraries…
Simulator

00-20-20

mean reward

mean reward

Results 3DoF: SCARA-40-60-40-60-80-80-100-1000123456(x105)

0200

400

time (s)iterations
Accuracy (mm)
SCARA
Simulated
Real

Repeatability (mm)1 Worker

8 Workers

1 Worker

8 Worker

2.80
26.14

1.37
12.89

4.52
28.96

3.57
10.83

800

1000

00-100-100

mean rewardmean reward

Results 6DoF: MARA-200-300-400-200-300-400-500-500-600-600

0

1234

56(x106)

01000

2000

time (s)iterations
Accuracy (mm)MARA
Simulated

3000

Repeatability (mm)1 Worker

8 Workers

1 Worker

8 Worker

0.05

0.08

0.02

0.02

4000

5000

a framework to accelerate
robot-training using
Gazebo and ROS in the cloudrobot_gym
cheaper• 12 replicas in parallel• 1603 hours of cloud computing:
0,134 e/hour per instance, or 1,606 e/hour for all the replicas
running at the same time

The total costs for all evaluations was 214 Eurosfasterevaluated 2 different modular robots
training-time can be reduced by more than 33%
while maintaining similar levels of accuracy

Perspectives and Future Worky-axis position (m)Robot trajectories under different levels of noise when reaching a target

tniojrepged68.
t2ni=oj
darr
ep
50g.
e0d
3
std
7.
5=
dar
1.
0std

x-axis position (m)Published Work- robot_gym: accelerated robot training through simulation in the cloud with ROS and Gazebo
https://arxiv.org/pdf/1808.10369.pdf- Towards self-adaptable robots: from programming to training machines
https://arxiv.org/pdf/1802.04082.pdf- Hierarchical Learning for Modular Robots
https://arxiv.org/pdf/1802.04132.pdf- Evaluation of Deep Reinforcement Learning Methods for Modular Robots
https://arxiv.org/pdf/1802.02395.pdf- gym_gazebo: a toolkit for reinforcement learning using ROS and Gazebo
https://arxiv.org/pdf/1608.05742.pdf

Check our webpage:
acutronicrobotics.com
Risto Kojcev
email: risto@erlerobotics.com