Package: dsrl
DSRL (Datasets for Safe Reinforcement Learning)provides a rich collection of datasets specifically designed for offline Safe Reinforcement Learning (RL). Created with the objective of fostering progress in offline safe RL research, DSRL bridges a crucial gap in the availability of safety-centric public benchmarks and datasets.DSRL provides:Diverse datasets:38 datasets across different safe RL environments and difficulty levels in SafetyGymnasium, BulletSafetyGym, and MetaDrive, all prepared with safety considerations.Consistent API with D4RL:For easy use and evaluation of offline learning methods.Data post-processing filters:Allowing alteration of data density, noise level, and reward distributions to simulate various data collection conditions.This package is a part of a comprehensive benchmarking suite that includesFSRLandOSRLand aims to promote advancements in the development and evaluation of safe learning algorithms.To learn more, please visit ourproject website.InstallationPull this repo and install:git clone https://github.com/liuzuxin/DSRL.git
cd DSRL
# install bullet_safety_gym only (by default)
pip install -e .
# install mujoco-based safety_gymnasium
pip install -e .[mujoco]
# install metadrive
pip install -e .[metadrive]
# install all in once
pip install -e .[all]How to use DSRLDSRL uses theGymnasiumAPI. Tasks are created via thegymnasium.makefunction. Each task is associated with a fixed offline dataset, which can be obtained with theenv.get_dataset()method. This method returns a dictionary with:observations: An N × obs_dim array of observations.next_observations: An N × obs_dim of next observations.actions: An N × act_dim array of actions.rewards: An N dimensional array of rewards.costs: An N dimensional array of costs.terminals: An N dimensional array of episode termination flags. This is true when episodes end due to termination conditions such as falling over.timeouts: An N dimensional array of termination flags. This is true when episodes end due to reaching the maximum episode length.The usage is similar toD4RL. Here is an example code:importgymnasiumasgymimportdsrl# Create the environmentenv=gym.make('OfflineCarCircle-v0')# Each task is associated with a dataset# dataset contains observations, next_observatiosn, actions, rewards, costs, terminals, timeoutsdataset=env.get_dataset()print(dataset['observations'])# An N x obs_dim Numpy array of observations# dsrl abides by the OpenAI gym interfaceobs,info=env.reset()obs,reward,terminal,timeout,info=env.step(env.action_space.sample())cost=info["cost"]# Apply dataset filters [optional]# dataset = env.pre_process_data(dataset, filter_cfgs)Datasets are automatically downloaded to the~/.dsrl/datasetsdirectory whenget_dataset()is called. If you would like to change the location of this directory, you can set the$DSRL_DATASET_DIRenvironment variable to the directory of your choosing, or pass in the dataset filepath directly into theget_datasetmethod.You can use run the following example scripts to play with the offline dataset of all the supported environments:pythonexamples/run_mujoco.py--agent[your_agent]--task[your_task]pythonexamples/run_bullet.py--agent[your_agent]--task[your_task]pythonexamples/run_metadrive.py--road[your_road]--traffic[your_traffic]Normalizing ScoresSet target cost by usingenv.set_target_cost(target_cost)function, wheretarget_costis the undiscounted sum of costs of an episodeYou can use theenv.get_normalized_score(return, cost_return)function to compute a normalized reward and cost for an episode, wherereturnsandcost_returnsare the undiscounted sum of rewards and costs respectively of an episode.The individual min and max reference returns are stored indsrl/infos.pyfor reference.LicenseAll datasets are licensed under theCreative Commons Attribution 4.0 License (CC BY), and code is licensed under theApache 2.0 License.