Package: offline-rl-ope
offline_rl_ope (BETA RELEASE)WARNINGWeighted importance sampling was incorrectly implemented in versions 1.X.X and 2.1.X, 2.2.XUnit testing currently only running in Python 3.11. 3.10 will be supported in the futureOnly 1 dimensional discrete action spaces are currently supported!IMPORTANT: THIS IS A BETA RELEASE. FUNCTIONALITY IS STILL BEING TESTEDFeedback/contributions are welcome :)Testing progresscomponents/ImportanceSampler.pyPolicy.pyOPEEstimationIS.pyutils.pyDirectMethod.py*DoublyRobust.pyMetricsEffectiveSampleSize.pyValidWeightsProp.pyPropensityModelsLowerBoundsapi/d3rlpyInsufficient functionality to test i.e., currently only wrapper classes are implemented for the OPEEstimation/DirectMethod.pyOverviewBasic unit testing has been implemented for all the core functionality of the package. The d3rlpy/api for importance sampling adds minimal additional functionality therefore, it is likely to function as expected however, no sepcific unit testing has been implemented!IMPORTANT:More documentation needs to be added however, please refer to examples/ for an illustration of the functionalityexamples/static.py provides an illustration of the package being used for evaluation post training. Whilst the d3rlpy package is used for model training, the script is agnostic to the evaluation model usedexamples/d3rlpy_training_api.py provides an illustration of how the package can be used to obtain incremental performance statistics during the training of d3rlpy models. It provides greater functionality to the native scorer metrics included in d3rlpyThe current focus has been on discrete action spaces. Continuous action spaces are intended to be addressed at a later dateDescriptionoffline_rl_ope aims to provide flexible and efficient implementations of OPE algorithms for use when training offline RL models. The main audience is researchers developing smaller, non-distributed models i.e., those who do not want to use packages such as ray (https://github.com/ray-project/ray).The majority of the first phase of development has been on efficient implementations of importance sampling and integration with d3rlpy.Main user assumptions of the projectUsers require smaller workloadsUsers should be able to easily implment new OPE methods using the importance sampling classes providedWhen developing models, users may want to run several different OPE methods e.g., per-decision IS, per-decision weight IS etc. These different methods fundamentally utilise the same set of IS weights. The library has been developed to efficiently cache these IS weights, significantly reducing the overhead of incremental OPE metrics whilst adding no overhead if only a single OPE method is required.OPE methodsImportance sampling:WeightedPer decisionClippedFitted Q-Evaluation (via d3rlpy)Doubly robust:WeightedPer decisionClippedLower boundsHCOPE (viahttps://github.com/gonzfe05/HCOPE/tree/master)BootstrappingCreditsCredit tohttps://github.com/takuseno/d3rlpyfor the implementation of FQECredit tohttps://github.com/gonzfe05/HCOPE/tree/masterfor the implementation of HCOPEInstallationPyPi:https://pypi.org/project/offline-rl-ope/To install from sourse using pip, clone this repository and runpip install .in the top level of the repo.Future workAsync/multithread supportAdditional estimators:DualDICEMAGICExtended DR estimator as per equation 12 inhttps://arxiv.org/pdf/1511.03722.pdfAPIsExtend d3rlpyAdd additional apis e.g. for stable baselinesContinuous action spacesTipsDebugging importance ratiosIf importance sampling based methods are evaluating to 0, consider visualising the importance ratios at different stages. All IS based estimators require an object of typesISWeightCalculatorto be defined (this could be aISWeightCalculatorobject or a subclass e.g.,ISWeightOrchestrator). In any case, visualising theis_weightsattribute of this object will provide insight regarding the course of NaN's.NaNvalues occur when the behaviour policy and evaluation policy have probability 0. In both cases a small epsilon value could be used in place of probability 0. For deterministic evaluation policies, this is automatically includedD3RlPyDeterministic.Infvalues can occur when the probability under the evaluation policy is greater than 0 whilst the behaviour policu probability is 0.The different kinds of importance samples can also be visualised by querying thetraj_is_weightsattribute of a givenImportanceSamplerobject. If for example, vanilla importance sampling is being used and the samples are notNaNorInfthen visualising thetraj_is_weightsmay provide insight. In particular, IS weights will tend to inifinity when the evaluation policy places large density on an action in comparison to the behaviour policy.Release log4.0.0Predefined propensity models including:Generic feedforward MLP for continuous and discrete action spaces built in PyTorchxGBoost for continuous and discrete action spaces built in sklearnBoth PyTorch and sklearn models can handle space discrete actions spaces i.e., a propensity model can be exposed to 'new' actions provided the full action space definition is provided at the training time of the propensity modelMetrics pattern with:Effective sample size calculationProportion of valid weights i.e., the mean proportion of weights between a min and max value across trajectoriesRefactored the BehavPolicy class to accept a 'policy_func' that aligns with the other policy classes3.0.33.10 support3.0.2PyPI release!Fixed bug in VanillaIS calculation where trajectories with less than the max number of samples were always being evaluated to 0Epsilon smoothing for deterministic evaluation policies3.0.1Updated d3rlpy API to align with the 2.x release!2.3.0Corrected error when calculating weighted importance samples. The weight was defined as the sum rather than the mean of time t ratiosImplemented smoothing for weighted importance sampling to prevents nan's when valid weights are 0.