Package: usienarl
This is apython 3.6and above library for Reinforcement Learning (RL) experiments.The idea behind this library is to generate an intuitive yet versatile system to generate RL agents, experiments, models, etc.
The library is modular, and allow for easy experiment iterations and validation. It is currently used in my research and it
was first built for that very purpose.The entire package system is vectorized using NumPy. This allows both the environments and the agents to run very fast,
especially on large scale experiments.Note:this is not meant to be an entry level framework. Users are expected to be at least somewhat experienced in both
python and reinforcement learning.FeaturesIncluded in package:Environment abstract class to wrap any environment with gym-like methodsInterface system to allow communication between agents and environment (for example to make a certain environment not fully observable)Customizable experiments, structured in training/validation and test volleys, plotting the following metrics:Average total reward (reward in one episode) over training, validation and test volleys and episodes (per each volley)Average scaled reward (reward per step) over training, validation and test volleys and episodes (per each volley)Standard deviation of said total and scaled rewards over training, validation and test volleysUtility functions to easily run experiments (even multiple iterations of one type of experiment)Many state-of-the-art algorithms, including:Tabular Temporal Difference Q-Learning, SARSA, Expected SARSA with Prioritized Experience Replay memory bufferDeep Temporal Difference Q-Learning (DQN), SARSA, Expected SARSA with Prioritized Experience Replay memory bufferDouble Deep Temporal Difference Q-Learning (DDQN) with Prioritized Experience Replay memory bufferDueling Temporal Difference Q-Learning (DDDQN) with Prioritized Experience Replay memory bufferVanilla Policy Gradient (VPG) with General Advantage Estimate (GAE) buffer using rewards-to-goProximal Policy Optimization (PPO) with General Advantage Estimate (GAE) buffer using rewards-to-go and early stoppingDeep Deterministic Policy Gradient (DDPG) with simple FIFO bufferDefault agents for all the included algorithms, including:Q-Learning both tabular and approximated by DNNs with Epsilon Greedy, Boltzmann and Dirichlet exploration policiesSARSA both tabular and approximated by DNNs with Epsilon Greedy, Boltzmann and Dirichlet exploration policiesExpected SARSA both tabular and approximated by DNNs with Epsilon Greedy, Boltzmann and Dirichlet exploration policiesVanilla Policy Gradient and Proximal Policy OptimizationDeep Deterministic Policy Gradient with Gaussian NoiseCustomizable config class to easily define the layers of the networks (when applicable)Not included in package:Extensive set of benchmarks for each default agent using the OpenAI gym environment as a referenceOpenAI gym environment wrappers and benchmark experiment as implementation samplesFor additional example of usage of this framework, take a look a these GitHub pages (using old versions of the framework):TicTacToeRLUltimateTicTacToeRLLicenseBSD 3-Clause LicenseFor additional information check the provided license file.How to installIf you only need to use the framework, just download the pip packageusienarland import the package in your scripts.When installing, make sure to choose the version suiting your computing capabilities.
If you have CUDA installed, the gpu version is advised. Otherwise, just use the cpu version.
To choose a version, specify your extra require during install:pip install usienarl[tensorflow-gpu] to install tensorflow-gpu versionpip install usienarl[tensorflow] to install tensorflow cpu versionNote:failure in specifying the extra require will cause tensorflow to not be installed, and as such the library won’t
be usable at all. For instance, this is not allowed,unless you already have tensorflow installed:pip install usienarlTensorflow support ranges from 1.10 to 1.15. Please report any kind of incompatibilities.
Some future warnings could be issued by tensorflow if they are not removed (look at the benchmarks to see how to remove them).If you want to improve/modify/extends the framework, or even just try my own benchmarks at home, download or clone
thegit repository.
You are welcome to open issues or participate in the project. Note that the benchmarks are usually run using tensorflow-gpu.RequirementsBesides Tensorflow, with this package also the following packages will be installed in your environment:NumPySciPyPandasMatplotlibHow to useFor a simple use case, refer to benchmark provided in therepository.
For advanced use, refer to the built-in documentation and to the provided source code in therepository.Current issuesDDPG is experimental. It should work but its performance is not consistent.
Experiment iterations are not implemented as well as they could be. I’m still thinking about a better implementation.Changelogv0.7.0:Vectorized the entire package.Added DDPG algorithm and agent.Improved all algorithms. They are now clearer, optimized and more intuitive in their implementation.Improved all default agents. They are now clearer, optimized and more intuitive in their implementation.Improved the experiment. Now volleys are classes and all collected data and metrics are easily accessible through attributes.Plots are now saved both per volley and per episode inside each volleyRefactored utility functionsA huge list of minor fixes and improvementsLargely improved summaries and built-in documentationv0.7.1:Fixed a return value of the tabular SARSA algorithmFixed interface now able to translate environment actions to agent actions on discrete spaces when possible actions list have different sizesAdded default routine to translate environment possible actions to agent possible actions (not fully optimized yet)Largely improved and optimized pass-through interface routine to translate between environment possible actions to agent possible actionsPossible actions are now list instead of arrays. Built-in documentation is updated accordingly.v0.7.2:Fixed some missing logs at the end of test in the experiment classFixed plots x-axis using sometimes float-values instead of int-valuesv0.7.3:Added a default random agentv0.7.4:Fixed a bug on tabular SARSA with dirichlet exploration policy on the transpose operationFixed a numerical warning on final result when the denominator could happen to be zerov0.7.5:Fixed a small bug in vanilla policy gradient and proximal policy optimization with continuous acton spacesv0.7.6:Updated the agent to be able to be restored when there is no need to save the model itselfv0.7.7:Updated the agent, the experiment and the episode volley to not require plots path and save path anymore when only inference (test) is runUpdated the experiment with a property to get the last test volleyUpdated the experiment and the episode volley with properties to keep track of all steps rewards for all executed volleys (of any type)Updated the experiment and the episode volley with properties to keep track of all episodes total and scaled rewards for all executed volleys (of any type)Updated the experiment and the episode volley with properties to keep track of all episodes lengths for all executed volleys (of any type)v0.7.8:Fixed a bug with reward computation when validating/testing when episode lengths are not the same across all episodesv0.7.9:Added time metrics for agent decision (it computes the time the agent requires to compute a set of actions)Now it’s possible to restore agents without checkpoint paths, to better support heuristicsv0.7.10:Fixed min advantage to clipped ratio to better respect original PPO paperCREDITSLuca Pasqualini atSAILab- University of Siena.