Package: rex-gym
Rex: an open-source quadruped robotThe goal of this project is to train an open-source 3D printed quadruped robot exploringReinforcement LearningandOpenAI Gym. The aim is to let the robot learns domestic and generic tasks in the simulations and then
successfully transfer the knowledge (Control Policies) on the real robot without any other manual tuning.This project is mostly inspired by the incredible works done by Boston Dynamics.Related repositoriesrexctl- A CLI application to bootstrap and control Rex running the trainedControl Policies.rex-cloud- A CLI application to train Rex on the cloud.Rex-gym: OpenAI Gym environments and toolsThis repository contains a collection ofOpenAI Gym Environmentsused to train Rex, the Rex URDF model,
the learning agent implementation (PPO) and some scripts to start the training session and visualise the learnedControl Polices.
This CLI application allows batch training, policy reproduction and single training rendered sessions.InstallationCreate aPython 3.7virtual environment, e.g. usingAnacondaconda create -n rex python=3.7 anaconda
conda activate rexPyPI packageInstall the publicrex-gympackage:pip install rex_gymInstall from sourceClone this repository and run from the root of the project:pip install .CLI usageRunrex-gym --helpto display the available commands andrex-gym COMMAND_NAME --helpto show the help
message for a specific command.Use the--argflag to eventually set the simulation arguments. For a full list check out theenvironments parameters.To switch between theOpen Loopand theBezier controller (inverse kinematics)modes, just append either the--open-loopor--inverse-kinematicsflags.rex-gym COMMAND_NAME -ik
rex-gym COMMAND_NAME -olFor more info about the modes check outthe learning approach.Policy player: run a pre-trained agentTo start a pre-trained agent (play a learnedControl Policy):rex-gym policy --env ENV_NAMETrain: Run a single training simulationTo start a single agent rendered session (agents=1,render=True):rex-gym train --playground True --env ENV_NAME --log-dir LOG_DIR_PATHTrain: Start a new batch training simulationTo start a new batch training session:rex-gym train --env ENV_NAME --log-dir LOG_DIR_PATHRobot platformMark 1The robot used for this first version is theSpotmicromade byDeok-yeon Kim.I've printed the components using a Creality Ender3 3D printer, with PLA and TPU+.The hardware used is listed in thiswiki.The idea is to extend the robot adding components like a robotic arm on the top of the rack and a LiDAR sensor in the next versions alongside
fixing some design issue to support a better (and easier) calibration and more reliable servo motors.Simulation modelBase modelRex is a 12 joints robot with 3 motors (Shoulder,LegandFoot) for each leg.The robotbasemodel is imported inpyBulletusing anURDF file.The servo motors are modelled in themodel/motor.pyclass.Robotic armThearmmodel has the open source 6DOF robotic armPoppy Ergo Jrequipped on the top of the
rack.To switch betweenbaseandarmmodels use the--markflag.Learning approachThis library uses theProximal Policy Optimization (PPO)algorithm with a hybrid policy defined as:a(t, o) = a(t) + π(o)It can be varied continuously from fully user-specified to entirely learned from scratch.
If we want to use a user-specified policy, we can set both the lower and the upper bounds ofπ(o)to be zero.
If we want a policy that is learned from scratch, we can seta(t) = 0and give the feedback componentπ(o)a wide output range.By varying the open loop signal and the output bound of the feedback component, we can decide how much user control is applied to the system.A twofold approach is used to implement the RexGym Environments:Bezier controllerandOpen Loop.TheBezier controllerimplements a fully user-specified policy. The controller uses theInverse Kinematicsmodel (seemodel/kinematics.py)
to generate the gait.TheOpen Loopmode consists, in some cases, in let the system lean from scratch (setting the open loop componenta(t) = 0) while others
just providing a simple trajectory reference (e.g.a(t) = sin(t)).The purpose is to compare the learned policies and scores using those two different approach.TasksThis is the list of tasks this experiment want to cover:Basic controls:Static poses - Frame a point standing on the spot.Bezier controllerOpen Loop signalGallopforwardBezier controllerOpen Loop signalbackwardBezier controllerOpen Loop signalWalkforwardBezier controllerOpen Loop signalbackwardBezier controllerOpen Loop signalTurn - on the spotBezier controllerOpen Loop signalStand up - from the floorBezier controllerOpen Loop signalNavigate uneven terrains:Random heightfield, hill, mountMazeStairsOpen a doorGrab an objectFall recoveryReach a specific point in a mapMap an open spaceTerrainsTo set a specific terrain, use the--terrainflag. The default terrain is the standardplane. This feature is quite useful to
test the policy robustness.Random heightfieldUse the--terrain randomflag to generate a random heighfield pattern. This pattern is updated at every 'Reset' step.HillsUse the--terrain hillsflag to generate an uneven terrain.MountsUse the--terrain mountsflag to generate this scenario.MazeUse the--terrain mazeflag to generate this scenario.EnvironmentsBasic Controls: Static posesGoal: Move Rex base to assume static poses standing on the spot.Inverse kinematicThe gym environment is used to learn how to gracefully assume a pose avoiding too fast transactions.
It uses a one-dimensionalaction spacewith a feedback componentπ(o)with bounds[-0.1, 0.1].
The feedback is applied to a sigmoid function to orchestrate the movement.
When the--playgroundflag is used, it's possible to use the pyBullet UI to manually set a specific pose altering the robot base position
(x,y,z) and orientation (roll,pitch,jaw).Basic Controls: GallopGoal: Gallop straight on and stop at a desired position.In order to make the learning more robust, the Rex target position is randomly chosen at every 'Reset' step.Bezier controllerThis gym environment is used to learn how to gracefully start the gait and then stop it after reaching the target position (on thexaxis).
It uses two-dimensionalaction spacewith a feedback componentπ(o)with bounds[-0.3, 0.3]. The feedback component is applied to two ramp functions
used to orchestrate the gait. A correct start contributes to void the drift effect generated by the gait in the resulted learned policy.Open Loop signalThis gym environment is used to let the system learn the gait from scratch. Theaction spacehas 4 dimensions, two for the front legs and feet
and two for the rear legs and feet, with the feedback component output bounds[−0.3, 0.3].Basic Controls: WalkGoal: Walk straight on and stop at a desired position.In order to make the learning more robust, the Rex target position is randomly chosen at every 'Reset' step.Bezier controllerThis gym environment is used to learn how to gracefully start the gait and then stop it after reaching the target position (on thexaxis).
It uses two-dimensionalaction spacewith a feedback componentπ(o)with bounds[-0.4, 0.4]. The feedback component is applied to two ramp functions
used to orchestrate the gait. A correct start contributes to void the drift effect generated by the gait in the resulted learned policy.ForwardBackwardsOpen Loop signalThis gym environment uses a sinusoidal trajectory reference to alternate the Rex legs during the gait.leg(t) = 0.1 cos(2π/T*t)
foot(t) = 0.2 cos(2π/T*t)The feedback component has very small bounds:[-0.01, 0.01]. A ramp function are used to start and stop the gait gracefully.Basic Controls: Turn on the spotGoal: Reach a target orientation turning on the spot.In order to make the learning more robust, the Rex start orientation and target are randomly chosen at every 'Reset' step.Bezier controllerThis gym environment is used to optimise thestep_lengthandstep_rotationarguments used by theGaitPlannerto implement the 'steer' gait.
It uses a two-dimensionalaction spacewith a feedback componentπ(o)with bounds[-0.05, 0.05].Open loopThis environment is used to learn a 'steer-on-the-spot' gait, allowing Rex to moving towards a specific orientation.
It uses a two-dimensionalaction spacewith a small feedback componentπ(o)with bounds[-0.05, 0.05]to optimise theshoulderandfootangles
during the gait.Basic Controls: Stand upGoal: Stand up starting from the standby position
This environment introduces therest_postion, ideally the position assumed when Rex is in standby.Open loopTheaction spaceis equals to 1 with a feedback componentπ(o)with bounds[-0.1, 0.1]used to optimise the signal timing.
The signal function applies a 'brake' forcing Rex to assume an halfway position before completing the movement.Environments parametersEnvironmentenvflagargflagGallopinggalloptarget_positionWalkingwalktarget_positionTurnturninit_orient,target_orientStand upstandupN.AargDescriptioninit_orientThe starting orientation in rad.target_orientThe target orientation in rad.target_positionThe target position (xaxis).FlagsDescriptionlog-dirThe path where the log directory will be created. (Required)playgroundA boolean to start a single training rendered sessionagents-numberSet the number of parallel agentsPPO Agent configurationYou may want to edit the PPO agent's default configuration, especially the number of parallel agents launched during
the simulation.Use the--agents-numberflag, e.g.--agents-number 10.This configuration will launch 10 agents (threads) in parallel to train your model.The default value is setup in theagents/scripts/configs.pyscript:def default():
    """Default configuration for PPO."""
    # General
    ...
    num_agents = 20CreditsPapersSim-to-Real: Learning Agile Locomotion For Quadruped Robotsand all the related papers. Google Brain, Google X, Google DeepMind - Minitaur Ghost Robotics.Inverse Kinematic Analysis Of A Quadruped RobotLeg Trajectory Planning for Quadruped Robots with High-Speed Trot GaitRobot platform v1Deok-yeon Kimcreator of SpotMini.The awesomePoppy Project.SpotMicro CAD files:SpotMicroAIcommunity.Inspiring projectsThe kinematics model was inspired by the great work done byMiguel Ayuso.