Package: ivy-gym
Fully differentiable reinforcement learning environments, written in Ivy.ContentsOverviewRun ThroughOptimization DemosGet InvoledOverviewWhat is Ivy Gym?Ivy Gym opens the door for intersectional research between supervised learning (SL), reinforcement learning (RL),
and trajectory optimization (TO),
by implementing RL environments in a fully differentiable manner.Specifically, Ivy gym provides differentiable implementations of the classic control tasks from OpenAI Gym,
as well as a new Swimmer task, which illustrates the simplicity of creating new tasks using Ivy.
The differentiable nature of the environments means that the cumulative reward can be directly optimized for in a supervised manner,
without need for reinforcement learning, which is the de facto approach for optimizing cumulative rewards. Check out thedocsfor more info!The library is built on top of the Ivy machine learning framework.
This means all environments simultaneously support:
Jax, Tensorflow, PyTorch, MXNet, and Numpy.Ivy LibrariesThere are a host of derived libraries written in Ivy, in the areas of mechanics, 3D vision, robotics, gym environments,
neural memory, pre-trained models + implementations, and builder tools with trainers, data loaders and more. Click on
the icons below to learn more!Quick StartIvy gym can be installed like so:pip installivy-gymTo quickly see the different environments provided, we suggest you check out thedemos!
We suggest you start by running the scriptrun_through.py,
and read the “Run Through” section below which explains this script.For demos which optimize performance on the different tasks, we suggest you run eitheroptimize_trajectory.pyoroptimize_policy.pyin theoptimizationdemos folder.Run ThroughThe different environemnts can be visualized via a simple script,
which executes random motion for 250 steps in one of the environments.
The script is available in thedemosfolder, as filerun_through.py.
First, we select a random backend framework to use for the examples, from the optionsivy.jax,ivy.tensorflow,ivy.torch,ivy.mxnetorivy.numpy,
and use this to set the ivy backend framework.importivyfromivy_demo_utils.framework_utilsimportchoose_random_frameworkivy.set_framework(choose_random_framework())We then select an environment to use and execute 250 random actions,
while rendering the environment after each step.By default, the demos all use theCartPoleenvironment, but this can be changed using the--envargument,
choosing from the optionsCartPole,Pendulum,MountainCar,ReacherorSwimmer.env=getattr(ivy_gym,env_str)()env.reset()ac_dim=env.action_space.shape[0]for_inrange(250):ac=ivy.random_uniform(-1,1,(ac_dim,))env.step(ac)env.render()Here, we briefly discuss each of the five environments,
before showing example episodes from a learnt policy network.
We use a learnt policy in these visualizations rather than random actions as used in the script,
because we find this to be more descriptive for visually explaining each task.
We also plot the instantaneous reward corresponding to each frame.CartPoleFor this task, a pole is attached by an un-actuated joint to a cart, which moves along a frictionless track.
The system is controlled by applying a force to the cart.
A reward is given based on the angle of the pendulum from being upright.
Example trajectories are given below.MountainCarFor this task, a car is on a one-dimensional track, positioned between two “mountains”.
The goal is to drive up the mountain on the right.
However, the car’s engine is not strong enough to scale the mountain in a single pass.
Therefore, the only way to succeed is to drive back and forth to build up momentum.
Here, the reward is greater if you spend less energy to reach the goal.
Example trajectories are given below.PendulumFor this task, an inverted pendulum starts in a random position, and the goal is to swing it up so it stays upright.
Again, a reward is given based on the angle of the pendulum from being upright.
Example trajectories are given below.ReacherFor this task, a 2-link robot arm must reach a target position.
Reward is given based on the distance of the end effector to the target.
Example trajectories are given below.SwimmerWe implemented this task ourselves, in order to highlight the simplicity of creating new custom environments.
For this task, a fish must swim to reach a target 2D positions whilst avoiding sharp obstacles.
Reward is given for being close to the target, and negative reward is given for colliding with the sharp objects.
Example trajectories are given below.Optimization DemosWe provide two demo scripts which optimize performance on these tasks in a supervised manner,
either via trajectory optimization or policy optimization.In the case of trajectory optimization, we optimize for a specific starting state of the environment,
whereas for policy optimization we train a policy network which is conditioned on the environment state,
and the starting state is then randomized between training steps.Rather than presenting the code here, we show visualizations of the demos.
The scripts for these demos can be found in theoptimizationdemos folder.Trajectory OptimizationIn this demo, we show trajectories on each of the five ivy gym environments during the course of trajectory optimization.
The optimization iteration is shown in the bottom right, along with the step in the environment.Policy OptimizationIn this demo, we show trajectories on each of the five ivy gym environments during the course of policy optimization.
The optimization iteration is shown in the bottom right, along with the step in the environment.Get InvoledWe hope the differentiable environments in this library are useful to a wide range of machine learning developers.
However, there are many more tasks which could be implemented.If there are any particular tasks you feel are missing,
or you would like to implement your own custom task,
then we are very happy to accept pull requests!We look forward to working with the community on expanding and improving the Ivy gym library.Citation@article{lenton2021ivy,
  title={Ivy: Unified Machine Learning for Inter-Framework Portability},
  author={Lenton, Daniel and Pardo, Fabio and Falck, Fabian and James, Stephen and Clark, Ronald},
  journal={arXiv preprint arXiv:2102.02886},
  year={2021}
}