Package: mdp-playground
MDP PlaygroundA python package to inject low-level dimensions of difficulties in RL environments. There are toy environments to design and debug RL agents. And complex environment wrappers for Atari and Mujoco to test robustness to these dimensions in complex environments.Getting startedThere are 4 parts to the package:Toy Environments: The base toy Environment inmdp_playground/envs/rl_toy_env.pyimplements the toy environment functionality, including discrete and continuous environments, and is parameterised by aconfigdict which contains all the information needed to instantiate the required MDP. Please seeexample.pyfor some simple examples of how to use the MDP environments in the package. For further details, please refer to the documentation inmdp_playground/envs/rl_toy_env.py.Complex Environment Wrappers: Similar to the toy environment, this is parameterised by aconfigdict which contains all the information needed to inject the dimensions into Atari or Mujoco environments. Please seeexample.pyfor some simple examples of how to use these. The Atari wrapper is inmdp_playground/envs/gym_env_wrapper.pyand the Mujoco wrapper is inmdp_playground/envs/mujoco_env_wrapper.py.Experiments: Experiments are launched usingrun_experiments.py. Config files for experiments are located inside theexperimentsdirectory. Please read theinstructionsbelow for details.Analysis:plot_experiments.ipynbcontains code to plot the standard plots from the paper.InstallationProduction useWe recommend usingcondato manage environments. After setup of the environment, you can install MDP Playground in two ways:ManualTo install MDP Playground manually, clone the repository and run:pipinstall-e.[extras]This might be the preferred way if you want easy access to the included experiments.From PyPIMDP Playground is also on PyPI. Just run:pipinstallmdp_playground[extras]Reproducing results from the paperWe recommend usingcondaenvironments to manage virtualPythonenvironments to run the experiments. Unfortunately, you will have to maintain 2 environments - 1 for the "older"discrete toyexperiments and 1 for the "newer"continuous and complexexperiments from the paper. As mentioned in Appendix P in the paper, this is because of issues with Ray, the library that we used for our baseline agents.Please follow the following commands to install for the discrete toy experiments:condacreate-npy36_toy_rl_disc_toypython=3.6
condaactivatepy36_toy_rl_disc_toycdmdp-playground
pipinstall-e.[extras_disc]Please follow the following commands to install for the continuous and complex experiments:condacreate-npy36_toy_rl_cont_comppython=3.6
condaactivatepy36_toy_rl_cont_compcdmdp-playground
pipinstall-e.[extras_cont]wget'https://ray-wheels.s3-us-west-2.amazonaws.com/master/8d0c1b5e068853bf748f72b1e60ec99d240932c6/ray-0.9.0.dev0-cp36-cp36m-manylinux1_x86_64.whl'pipinstallray-0.9.0.dev0-cp36-cp36m-manylinux1_x86_64.whl[rllib,debug]Running experimentsFor reproducing experiments from the main paper, please seebelow.For general instructions, please continue reading.You can run experiments using:run-mdpp-experiments -c <config_file> -e <exp_name> -n <config_num>Theexp_nameis a prefix for the filenames of CSV files where stats for the experiments are recorded. The CSV stats files will be saved to the current directory.Each of the command line arguments has defaults. Please refer to the documentation insiderun_experiments.pyfor further details on the command line arguments. (Or run it with the-hflag to bring up help.)The config files for experiments from thepaperare in the experiments directory.The name of the file corresponding to an experiment is formed as:<algorithm_name>_<dimension_names>.pySome samplealgorithm_names are:dqn,rainbow,a3c,a3c_lstm,ddpg,td3andsacSome sampledimension_names are:seq_del(fordelayandsequence lengthvaried together),p_r_noises(forPandR noisesvaried together),target_radius(for varyingtarget radius) andtime_unit(for varyingtime unit)For example, for algorithmDQNwhen varying dimensionsdelayandsequence length, the corresponding experiment file isdqn_seq_del.pyRunning experiments from the main paperWe list here the commands for the experiments from the main paper:# Discrete toy environments:# Image representation experiments:condaactivatepy36_toy_rl_disc_toy
pythonrun_experiments.py-cexperiments/dqn_image_representations.py-edqn_image_representations
pythonrun_experiments.py-cexperiments/rainbow_image_representations.py-erainbow_image_representations
pythonrun_experiments.py-cexperiments/a3c_image_representations.py-ea3c_image_representations
pythonrun_experiments.py-cexperiments/dqn_image_representations_sh_quant.py-edqn_image_representations_sh_quant# Continuous toy environments:condaactivatepy36_toy_rl_cont_comp
pythonrun_experiments.py-cexperiments/ddpg_move_to_a_point_time_unit.py-eddpg_move_to_a_point_time_unit
pythonrun_experiments.py-cexperiments/ddpg_move_to_a_point_irr_dims.py-eddpg_move_to_a_point_irr_dims# Varying the action range and time unit together for transition_dynamics_order = 2pythonrun_experiments.py-cexperiments/ddpg_move_to_a_point_p_order_2.py-eddpg_move_to_a_point_p_order_2# Complex environments:condaactivatepy36_toy_rl_cont_comp
pythonrun_experiments.py-cexperiments/dqn_qbert_del.py-edqn_qbert_del
pythonrun_experiments.py-cexperiments/ddpg_halfcheetah_time_unit.py-eddpg_halfcheetah_time_unit# For the spider plots, experiments for all the agents and dimensions will need to be run from the experiments directory, i.e., for discrete: dqn_p_r_noises.py, a3c_p_r_noises, ..., dqn_seq_del, ..., dqn_sparsity, ..., dqn_image_representations, ...# for continuous:, ddpg_move_to_a_point_p_noise, td3_move_to_a_point_p_noise, ..., ddpg_move_to_a_point_r_noise, ..., ddpg_move_to_a_point_irr_dims, ..., ddpg_move_to_a_point_action_loss_weight, ..., ddpg_move_to_a_point_action_max, ..., ddpg_move_to_a_point_target_radius, ..., ddpg_move_to_a_point_time_unit# and then follow the instructions in plot_experiments.ipynb# For the bsuite debugging experiment, please run the bsuite sonnet dqn agent on our toy environment while varying reward density. Commit https://github.com/deepmind/bsuite/commit/5116216b62ce0005100a6036fb5397e358652530 should work fine.The CSV stats files will be saved to the current directory and can be analysed inplot_experiments.ipynb.PlottingTo plot results from experiments, runjupyter-notebookand openplot_experiments.ipynbin Jupyter. There are instructions within each of the cells on how to generate and save plots.CitingIf you use MDP Playground in your work, please cite the following paper:@article{rajan2020mdp,title={MDP Playground: Controlling Dimensions of Hardness in Reinforcement Learning},author={Raghu Rajan and Jessica Lizeth Borja Diaz and Suresh Guttikonda and Fabio Ferreira and André Biedenkapp and Frank Hutter},year={2020},eprint={1909.07750},archivePrefix={arXiv},primaryClass={cs.LG}}