Package: yarlp
|Build Status|yarlp-----**Yet Another Reinforcement Learning Package**Implementations of ```CEM`` </yarlp/agent/cem_agent.py>`__,```REINFORCE`` </yarlp/agent/pg_agents.py>`__,```TRPO`` </yarlp/agent/trpo_agent.py>`__,```DDQN`` </yarlp/agent/ddqn_agent.py>`__,```A2C`` </yarlp/agent/a2c_agent.py>`__ with reproducible benchmarks.Experiments are templated using ``jsonschema`` and are compared topublished results. This is meant to be a starting point for workingimplementations of classic RL algorithms. Unfortunately evenimplementations from OpenAI baselines are `not alwaysreproducible <https://github.com/openai/baselines/issues/176>`__.A working Dockerfile with ``yarlp`` installed can be run with:-  ``docker build -t "yarlpd" .``-  ``docker run -it yarlpd bash``To run a benchmark, simply:``python yarlp/experiment/experiment.py --help``If you want to run things manually, look in ``examples`` or look atthis:.. code:: pythonfrom yarlp.agent.trpo_agent import TRPOAgentfrom yarlp.utils.env_utils import NormalizedGymEnvenv = NormalizedGymEnv('MountainCarContinuous-v0')agent = TRPOAgent(env, seed=123)agent.train(max_timesteps=1000000)Benchmarks----------We benchmark against published results and Openai```baselines`` <https://github.com/openai/baselines>`__ where availableusing```yarlp/experiment/experiment.py`` </yarlp/experiment/experiment.py>`__.Benchmark scripts for Openai ``baselines`` were made ad-hoc, such as`thisone <https://github.com/btaba/baselines/blob/master/baselines/trpo_mpi/run_trpo_experiment.py>`__.Atari10M~~~~~~~~+---------------+--------------+-------------------+| |BeamRider|   | |Breakout|   | |Pong|            |+---------------+--------------+-------------------+| |QBert|       | |Seaquest|   | |SpaceInvaders|   |+---------------+--------------+-------------------+DDQN with dueling networks and prioritized replay^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^``python yarlp/experiment/experiment.py run_atari10m_ddqn_benchmark``I trained 6 Atari environments for 10M time-steps (**40M frames**),using 1 random seed, since I only have 1 GPU and limited time on thisEarth. I used DDQN with dueling networks, but no prioritized replay(although it's implemented). I compare the final mean 100 episode rawscores for yarlp (with exploration of 0.01) with results from `Hasseltet al, 2015 <https://arxiv.org/pdf/1509.06461.pdf>`__ and `Wang et al,2016 <https://arxiv.org/pdf/1511.06581.pdf>`__ which train for **200Mframes** and evaluate on 100 episodes (exploration of 0.05).I don't compare to OpenAI baselines because the OpenAI DDQNimplementation is **not** currently able to reproduce published resultsas of 2018-01-20. See `this githubissue <https://github.com/openai/baselines/issues/176>`__, although Ifound `these benchmarkplots <https://github.com/openai/baselines-results/blob/master/dqn_results.ipynb>`__to be pretty helpful.+------+------+------+------+| env  | yarl | Hass | Wang ||      | p    | elt  | et   ||      | DUEL | et   | al   ||      | 40M  | al   | DUEL ||      | Fram | DDQN | 200M ||      | es   | 200M | Fram ||      |      | Fram | es   ||      |      | es   |      |+======+======+======+======+| Beam | 8705 | 7654 | 1216 || Ride |      |      | 4    || r    |      |      |      |+------+------+------+------+| Brea | 423. | 375  | 345  || kout | 5    |      |      |+------+------+------+------+| Pong | 20.7 | 21   | 21   ||      | 3    |      |      |+------+------+------+------+| QBer | 5410 | 1487 | 1922 || t    | .75  | 5    | 0.3  |+------+------+------+------+| Seaq | 5300 | 7995 | 5024 || uest | .5   |      | 5.2  |+------+------+------+------+| Spac | 1978 | 3154 | 6427 || eInv | .2   | .6   | .3   || ader |      |      |      || s    |      |      |      |+------+------+------+------++------+------+------+------+| |Bea | |Bre | |Pon | |Qbe || mRid | akou | gNoF | rtNo || erNo | tNoF | rame | Fram || Fram | rame | skip | eski || eski | skip | -v4| | p-v4 || p-v4 | -v4| |      | |    || |    |      |      |      |+------+------+------+------+| |Sea | |Spa |      |      || ques | ceIn |      |      || tNoF | vade |      |      || rame | rsNo |      |      || skip | Fram |      |      || -v4| | eski |      |      ||      | p-v4 |      |      ||      | |    |      |      |+------+------+------+------+A2C^^^``python yarlp/experiment/experiment.py run_atari10m_a2c_benchmark``A2C on 10M time-steps (**40M frames**) with 1 random seed. Resultscompared to learning curves from `Mnih et al,2016 <https://arxiv.org/pdf/1602.01783.pdf>`__ extracted at 10Mtime-steps from Figure 3. You are invited to run for multiple seeds andthe full 200M frames for a better comparison.+-----------------+-----------------+---------------------------------+| env             | yarlp A2C 40M   | Mnih et al A3C 40M 16-threads   |+=================+=================+=================================+| BeamRider       | 3150            | ~3000                           |+-----------------+-----------------+---------------------------------+| Breakout        | 418             | ~150                            |+-----------------+-----------------+---------------------------------+| Pong            | 20              | ~20                             |+-----------------+-----------------+---------------------------------+| QBert           | 3644            | ~1000                           |+-----------------+-----------------+---------------------------------+| SpaceInvaders   | 805             | ~600                            |+-----------------+-----------------+---------------------------------++------+------+------+------+| |Bea | |Bre | |Pon | |Qbe || mRid | akou | gNoF | rtNo || erNo | tNoF | rame | Fram || Fram | rame | skip | eski || eski | skip | -v4| | p-v4 || p-v4 | -v4| |      | |    || |    |      |      |      |+------+------+------+------+| |Sea | |Spa |      |      || ques | ceIn |      |      || tNoF | vade |      |      || rame | rsNo |      |      || skip | Fram |      |      || -v4| | eski |      |      ||      | p-v4 |      |      ||      | |    |      |      |+------+------+------+------+Here are some `moreplots <https://github.com/openai/baselines-results/blob/master/acktr_ppo_acer_a2c_atari.ipynb>`__from OpenAI to compare against.Mujoco1M~~~~~~~~TRPO^^^^``python yarlp/experiment/experiment.py run_mujoco1m_benchmark``We average over 5 random seeds instead of 3 for both ``baselines`` and``yarlp``. More seeds probably wouldn't hurt here, we report 95thpercent confidence intervals.+-------------------------------+--------------------+-------------------------+----------------+| |Hopper-v1|                   | |HalfCheetah-v1|   | |Reacher-v1|            | |Swimmer-v1|   |+-------------------------------+--------------------+-------------------------+----------------+| |InvertedDoublePendulum-v1|   | |Walker2d-v1|      | |InvertedPendulum-v1|   |                |+-------------------------------+--------------------+-------------------------+----------------+CLI scripts-----------CLI convenience scripts will be installed with the package:-  Run a benchmark:-  ``python yarlp/experiment/experiment.py --help``-  Plot ``yarlp`` compared to Openai ``baselines`` benchmarks:-  ``compare_benchmark <yarlp-experiment-dir> <baseline-experiment-dir>``-  Experiments:-  Experiments can be defined using json, validated with``jsonschema``. See `here </experiment_configs>`__ for sampleexperiment configs. You can do a grid search if multipleparameters are specified, which will run in parallel.-  Example:``run_yarlp_experiment --spec-file experiment_configs/trpo_experiment_mult_params.json``-  Experiment plots:-  ``make_plots <experiment-dir>``.. |Build Status| image:: https://travis-ci.org/btaba/yarlp.svg?branch=master:target: https://travis-ci.org/btaba/yarlp.. |BeamRider| image:: /assets/atari10m/ddqn/beamrider.gif.. |Breakout| image:: /assets/atari10m/ddqn/breakout.gif.. |Pong| image:: /assets/atari10m/ddqn/pong.gif.. |QBert| image:: /assets/atari10m/ddqn/qbert.gif.. |Seaquest| image:: /assets/atari10m/ddqn/seaquest.gif.. |SpaceInvaders| image:: /assets/atari10m/ddqn/spaceinvaders.gif.. |BeamRiderNoFrameskip-v4| image:: /assets/atari10m/ddqn/BeamRiderNoFrameskip-v4.png.. |BreakoutNoFrameskip-v4| image:: /assets/atari10m/ddqn/BreakoutNoFrameskip-v4.png.. |PongNoFrameskip-v4| image:: /assets/atari10m/ddqn/PongNoFrameskip-v4.png.. |QbertNoFrameskip-v4| image:: /assets/atari10m/ddqn/QbertNoFrameskip-v4.png.. |SeaquestNoFrameskip-v4| image:: /assets/atari10m/ddqn/SeaquestNoFrameskip-v4.png.. |SpaceInvadersNoFrameskip-v4| image:: /assets/atari10m/ddqn/SpaceInvadersNoFrameskip-v4.png.. |BeamRiderNoFrameskip-v4| image:: /assets/atari10m/a2c/BeamRiderNoFrameskip-v4.png.. |BreakoutNoFrameskip-v4| image:: /assets/atari10m/a2c/BreakoutNoFrameskip-v4.png.. |PongNoFrameskip-v4| image:: /assets/atari10m/a2c/PongNoFrameskip-v4.png.. |QbertNoFrameskip-v4| image:: /assets/atari10m/a2c/QbertNoFrameskip-v4.png.. |SeaquestNoFrameskip-v4| image:: /assets/atari10m/a2c/SeaquestNoFrameskip-v4.png.. |SpaceInvadersNoFrameskip-v4| image:: /assets/atari10m/a2c/SpaceInvadersNoFrameskip-v4.png.. |Hopper-v1| image:: /assets/mujoco1m/trpo/Hopper-v1.png.. |HalfCheetah-v1| image:: /assets/mujoco1m/trpo/HalfCheetah-v1.png.. |Reacher-v1| image:: /assets/mujoco1m/trpo/Reacher-v1.png.. |Swimmer-v1| image:: /assets/mujoco1m/trpo/Swimmer-v1.png.. |InvertedDoublePendulum-v1| image:: /assets/mujoco1m/trpo/InvertedDoublePendulum-v1.png.. |Walker2d-v1| image:: /assets/mujoco1m/trpo/Walker2d-v1.png.. |InvertedPendulum-v1| image:: /assets/mujoco1m/trpo/InvertedPendulum-v1.png