Package: rl-games
RL Games: High performance RL libraryDiscord Channel Linkhttps://discord.gg/hnYRq7DsQhPapers and related linksIsaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning:https://arxiv.org/abs/2108.10470DeXtreme: Transfer of Agile In-Hand Manipulation from Simulation to Reality:https://dextreme.org/https://arxiv.org/abs/2210.13702Transferring Dexterous Manipulation from GPU Simulation to a Remote Real-World TriFinger:https://s2r2-ig.github.io/https://arxiv.org/abs/2108.09779Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?https://arxiv.org/abs/2011.09533Superfast Adversarial Motion Priors (AMP) implementation:https://twitter.com/xbpeng4/status/1506317490766303235https://github.com/NVIDIA-Omniverse/IsaacGymEnvsOSCAR: Data-Driven Operational Space Control for Adaptive and Robust Robot Manipulation:https://cremebrule.github.io/oscar-web/https://arxiv.org/abs/2110.00704EnvPool: A Highly Parallel Reinforcement Learning Environment Execution Engine:https://arxiv.org/abs/2206.10558andhttps://github.com/sail-sg/envpoolTimeChamber: A Massively Parallel Large Scale Self-Play Framework:https://github.com/inspirai/TimeChamberSome results on the different environmentsNVIDIA Isaac GymDextremeDexPBTStarcraft 2 Multi AgentsBRAXMujoco EnvpoolDeepMind EnvpoolAtari EnvpoolRandom EnvsImplemented in Pytorch:PPO with the support of asymmetric actor-critic variantSupport of end-to-end GPU accelerated training pipeline with Isaac Gym and BraxMasked actions supportMulti-agent training, decentralized and centralized critic variantsSelf-playImplemented in Tensorflow 1.x (was removed in this version):Rainbow DQNA2CPPOQuickstart: Colab in the CloudExplore RL Games quick and easily in colab notebooks:Mujoco trainingMujoco envpool training example.Brax trainingBrax training example, with keeping all the observations and actions on GPU.Onnx discrete space export example with Cartpoleenvpool training example.Onnx continuous space export example with Pendulumenvpool training example.Onnx continuous space with LSTM export example with Pendulumenvpool training example.InstallationFor maximum training performance a preliminary installation of Pytorch 1.9+ with CUDA 11.1+ is highly recommended:conda install pytorch torchvision cudatoolkit=11.3 -c pytorch -c nvidiaor:pip install torch==1.11.0+cu113 torchvision==0.12.0+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.htmlThen:pip install rl-gamesTo run CPU-based environments either Ray or envpool are requiredpip install envpoolorpip install rayTo run Mujoco, Atari games or Box2d based environments training they need to be additionally installed withpip install gym[mujoco],pip install gym[atari]orpip install gym[box2d]respectively.To run Atari alsopip install opencv-pythonis required. In addition installation of envpool for maximum simulation and training perfromance of Mujoco and Atari environments is highly recommended:pip install envpoolCitingIf you use rl-games in your research please use the following citation:@misc{rl-games2021,title={rl-games: A High-performance Framework for Reinforcement Learning},author={Makoviichuk, Denys and Makoviychuk, Viktor},month={May},year={2021},publisher={GitHub},journal={GitHub repository},howpublished={\url{https://github.com/Denys88/rl_games}},}Development setuppoetryinstall# install cuda related dependenciespoetryrunpipinstalltorch==1.11.0+cu113torchvision==0.12.0+cu113-fhttps://download.pytorch.org/whl/cu113/torch_stable.htmlTrainingNVIDIA Isaac GymDownload and follow the installation instructions of Isaac Gym:https://developer.nvidia.com/isaac-gymAnd IsaacGymEnvs:https://github.com/NVIDIA-Omniverse/IsaacGymEnvsAntpython train.py task=Ant headless=Truepython train.py task=Ant test=True checkpoint=nn/Ant.pth num_envs=100Humanoidpython train.py task=Humanoid headless=Truepython train.py task=Humanoid test=True checkpoint=nn/Humanoid.pth num_envs=100Shadow Hand block orientation taskpython train.py task=ShadowHand headless=Truepython train.py task=ShadowHand test=True checkpoint=nn/ShadowHand.pth num_envs=100OtherAtari Pongpoetryinstall-Eatari
poetryrunpythonrunner.py--train--filerl_games/configs/atari/ppo_pong.yaml
poetryrunpythonrunner.py--play--filerl_games/configs/atari/ppo_pong.yaml--checkpointnn/PongNoFrameskip.pthBrax Antpoetryinstall-Ebrax
poetryrunpipinstall--upgrade"jax[cuda]==0.3.13"-fhttps://storage.googleapis.com/jax-releases/jax_releases.html
poetryrunpythonrunner.py--train--filerl_games/configs/brax/ppo_ant.yaml
poetryrunpythonrunner.py--play--filerl_games/configs/brax/ppo_ant.yaml--checkpointruns/Ant_brax/nn/Ant_brax.pthExperiment trackingrl_games support experiment tracking withWeights and Biases.poetryinstall-Eatari
poetryrunpythonrunner.py--train--filerl_games/configs/atari/ppo_breakout_torch.yaml--trackWANDB_API_KEY=xxxxpoetryrunpythonrunner.py--train--filerl_games/configs/atari/ppo_breakout_torch.yaml--track
poetryrunpythonrunner.py--train--filerl_games/configs/atari/ppo_breakout_torch.yaml--wandb-project-namerl-games-special-test--track
poetryrunpythonrunner.py--train--filerl_games/configs/atari/ppo_breakout_torch.yaml--wandb-project-namerl-games-special-test-wandb-entityopenrlbenchmark--trackMulti GPUWe usetorchrunto orchestrate any multi-gpu runs.torchrun--standalone--nnodes=1--nproc_per_node=2runner.py--train--filerl_games/configs/ppo_cartpole.yamlConfig ParametersFieldExample ValueDefaultDescriptionseed8NoneSeed for pytorch, numpy etc.algoAlgorithm block.namea2c_continuousNoneAlgorithm name. Possible values are: sac, a2c_discrete, a2c_continuousmodelModel block.namecontinuous_a2c_logstdNonePossible values: continuous_a2c ( expects sigma to be (0, +inf), continuous_a2c_logstd  ( expects sigma to be (-inf, +inf), a2c_discrete, a2c_multi_discretenetworkNetwork description.nameactor_criticPossible values: actor_critic or soft_actor_critic.separateFalseWhether use or not separate network with same same architecture for critic. In almost all cases if you normalize value it is better to have it FalsespaceNetwork spacecontinuouscontinuous or discretemu_activationNoneActivation for mu. In almost all cases None works the best, but we may try tanh.sigma_activationNoneActivation for sigma. Will be threated as log(sigma) or sigma depending on model.mu_initInitializer for mu.namedefaultsigma_initInitializer for sigma. if you are using logstd model good value is 0.nameconst_initializerval0fixed_sigmaTrueIf true then sigma vector doesn't depend on input.cnnConvolution block.typeconv2dType: right now two types supported: conv2d or conv1dactivationeluactivation between conv layers.initializerInitialier. I took some names from the tensorflow.nameglorot_normal_initializerInitializer namegain1.4142Additional parameter.convsConvolution layers. Same parameters as we have in torch.filters32Number of filters.kernel_size8Kernel size.strides4Stridespadding0Paddingfilters64Next convolution layer info.kernel_size4strides2padding0filters64kernel_size3strides1padding0mlpMLP Block. Convolution is supported too. See other config examples.unitsArray of sizes of the MLP layers, for example: [512, 256, 128]d2rlFalseUse d2rl architecture fromhttps://arxiv.org/abs/2010.09163.activationeluActivations between dense layers.initializerInitializer.namedefaultInitializer name.rnnRNN block.namelstmRNN Layer name. lstm and gru are supported.units256Number of units.layers1Number of layersbefore_mlpFalseFalseApply rnn before mlp block or not.configRL Config block.reward_shaperReward Shaper. Can apply simple transformations.min_val-1You can apply min_val, max_val, scale and shift.scale_value0.11normalize_advantageTrueTrueNormalize Advantage.gamma0.995Reward Discounttau0.95Lambda for GAE. Called tau by mistake long time ago because lambda is keyword in python :(learning_rate3e-4Learning rate.namewalkerName which will be used in tensorboard.save_best_after10How many epochs to wait before start saving checkpoint with best score.score_to_win300If score is >=value then this value training will stop.grad_norm1.5Grad norm. Applied if truncate_grads is True. Good value is in (1.0, 10.0)entropy_coef0Entropy coefficient. Good value for continuous space is 0. For discrete is 0.02truncate_gradsTrueApply truncate grads or not. It stabilizes training.env_nameBipedalWalker-v3Envinronment name.e_clip0.2clip parameter for ppo loss.clip_valueFalseApply clip to the value loss. If you are using normalize_value you don't need it.num_actors16Number of running actors/environments.horizon_length4096Horizon length per each actor. Total number of steps will be num_actors*horizon_length * num_agents (if env is not MA num_agents==1).minibatch_size8192Minibatch size. Total number number of steps must be divisible by minibatch size.minibatch_size_per_env8Minibatch size per env. If specified will overwrite total number number the default minibatch size with minibatch_size_per_env * nume_envs value.mini_epochs4Number of miniepochs. Good value is in [1,10]critic_coef2Critic coef. by default critic_loss = critic_coef * 1/2 * MSE.lr_scheduleadaptiveNoneScheduler type. Could be None, linear or adaptive. Adaptive is the best for continuous control tasks. Learning rate is changed changed every miniepochkl_threshold0.008KL threshould for adaptive schedule. if KL < kl_threshold/2 lr = lr * 1.5 and opposite.normalize_inputTrueApply running mean std for input.bounds_loss_coef0.0Coefficient to the auxiary loss for continuous space.max_epochs10000Maximum number of epochs to run.max_frames5000000Maximum number of frames (env steps) to run.normalize_valueTrueUse value running mean std normalization.use_diagnosticsTrueAdds more information into the tensorboard.value_bootstrapTrueBootstraping value when episode is finished. Very useful for different locomotion envs.bound_loss_typeregularisationNoneAdds aux loss for continuous case. 'regularisation' is the sum of sqaured actions. 'bound' is the sum of actions higher than 1.1.bounds_loss_coef0.00050Regularisation coefficientuse_smooth_clampFalseUse smooth clamp instead of regular for clipingzero_rnn_on_doneFalseTrueIf False RNN internal state is not reset (set to 0) when an environment is rest. Could improve training in some cases, for example when domain randomization is onplayerPlayer configuration block.renderTrueFalseRender environmentdeterministicTrueTrueUse deterministic policy ( argmax or mu) or stochastic.use_vecenvTrueFalseUse vecenv to create environment for playergames_num200Number of games to run in the player mode.env_configEnv configuration block. It goes directly to the environment. This example was take for my atari wrapper.skip4Number of frames to skipnameBreakoutNoFrameskip-v4The exact name of an (atari) gym env. An example, depends on the training env this parameters can be different.evaluationTrueFalseEnables the evaluation feature for inferencing while training.update_checkpoint_freq100100Frequency in number of steps to look for new checkpoints.dir_to_monitorDirectory to search for checkpoints in during evaluation.Custom network example:simple test networkThis network takes dictionary observation.
To register it you can add code in yourinit.pyfrom rl_games.envs.test_network import TestNetBuilder 
from rl_games.algos_torch import model_builder
model_builder.register_network('testnet', TestNetBuilder)simple test environmentexample environmentAdditional environment supported properties and functionsFieldDefault ValueDescriptionuse_central_valueFalseIf true than returned obs is expected to be dict with 'obs' and 'state'value_size1Shape of the returned rewards. Network wil support multihead value automatically.concat_infosFalseShould default vecenv convert list of dicts to the dicts of lists. Very usefull if you want to use value_boostrapping. in this case you need to always return 'time_outs' : True or False, from the env.get_number_of_agents(self)1Returns number of agents in the environmenthas_action_mask(self)FalseReturns True if environment has invalid actions mask.get_action_mask(self)NoneReturns action masks if  has_action_mask is true.  Good example isSMAC EnvRelease Notes1.6.1 (Unreleased)Fixed Central Value RNN bug which occurs if you train ma multi agent environment.Added Deepmind Control PPO benchmark.Added a few more experimental ways to train value prediction (OneHot, TwoHot encoding and crossentropy loss instead of L2).New methods didn't. It is impossible to turn it on from the yaml files. Once we find an env which trains better it will be added to the config.Added shaped reward graph to the tensorboard.Fixed bug with SAC not saving weights with save_frequency.Added multi-node training support for GPU-accelerated training environments like Isaac Gym. No changes in training scripts are required. Thanks to @ankurhanda and @ArthurAllshire for assistance in implementation.Added evaluation feature for inferencing during training. Checkpoints from training process can be automatically picked up and updated in the inferencing process when enabled.Added get/set API for runtime update of rl training parameters. Thanks to @ArthurAllshire for the initial version of fast PBT code.Fixed SAC not loading weights properly.Removed Ray dependency for use cases it's not required.Added warning for using deprecated 'seq_len' instead of 'seq_length' in configs with RNN networks.1.6.0Added ONNX export colab example for discrete and continious action spaces. For continuous case LSTM policy example is provided as well.Improved RNNs training in continuous space, added optionzero_rnn_on_done.Added NVIDIA CuLE support:https://github.com/NVlabs/culeAdded player config everride. Vecenv is used for inference.Fixed multi-gpu training with central value.Fixed max_frames termination condition, and it's interaction with the linear learning rate:https://github.com/Denys88/rl_games/issues/212Fixed "deterministic" misspelling issue.Fixed Mujoco and Brax SAC configs.Fixed multiagent envs statistics reporting. Fixed Starcraft2 SMAC environments.1.5.2Added observation normalization to the SAC.Returned back adaptive KL legacy mode.1.5.1Fixed build package issue.1.5.0Added wandb support.Added poetry support.Fixed various bugs.Fixed cnn input was not divided by 255 in case of the dictionary obs.Added more envpool mujoco and atari training examples. Some of the results: 15 min Mujoco humanoid training, 2 min atari pong.Added Brax and Mujoco colab training examples.Added 'seed' command line parameter. Will override seed in config in case it's > 0.Deprecatedhorovodin favor oftorch.distributed(#171).1.4.0Added discord channelhttps://discord.gg/hnYRq7DsQh:)Added envpool support with a few atari examples. Works 3-4x time faster than ray.Added mujoco results. Much better than openai spinning up ppo results.Added tcnn(https://github.com/NVlabs/tiny-cuda-nn) support. Reduces 5-10% of training time in the IsaacGym envs.Various fixes and improvements.1.3.2Added 'sigma' command line parameter. Will override sigma for continuous space in case if fixed_sigma is True.1.3.1Fixed SAC not working1.3.0Simplified rnn implementation. Works a little bit slower but much more stable.Now central value can be non-rnn if policy is rnn.Removed load_checkpoint from the yaml file. now --checkpoint works for both train and play.1.2.0Added Swish (SILU) and GELU activations, it can improve Isaac Gym results for some of the envs.Removed tensorflow and made initial cleanup of the old/unused code.Simplified runner.Now networks are created in the algos with load_network method.1.1.4Fixed crash in a play (test) mode in player, when simulation and rl_devices are not the same.Fixed variuos multi gpu errors.1.1.3Fixed crash when running single Isaac Gym environment in a play (test) mode.Added config parameterclip_actionsfor switching off internal action clipping and rescaling1.1.0Added to pypi:pip install rl-gamesAdded reporting env (sim) step fps, without policy inference. Improved naming.Renames in yaml config for better readability: steps_num to horizon_length amd lr_threshold to kl_thresholdTroubleshoutingSome of the supported envs are not installed with setup.py, you need to manually install themStarting from rl-games 1.1.0 old yaml configs won't be compatible with the new version:steps_numshould be changed tohorizon_lengthamdlr_thresholdtokl_thresholdKnown issuesRunning a single environment with Isaac Gym can cause crash, if it happens switch to at least 2 environments simulated in parallel