Repository: DLR-RM/rl-baselines3-zoo (language: python)

Issue:
[feature request]Add Mujoco-Gym since mujoco is free now
Hi,

I would ask that if you would like to add support of mujoco gym like Ant-v2..., before it's not possible since it's commercial software but now mujoco is free now.

best regards.


Discussion / hints:
Hello,
the answer is yes but I would also welcome help for that ;)

Fix patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 6921609d3..5a5f01286 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,3 +1,19 @@
+## Release 1.3.1a2 (WIP)
+
+### Breaking Changes
+
+
+### New Features
+- Added mujoco hyperparameters
+- Added MuJoCo pre-trained agents
+
+### Bug fixes
+
+### Documentation
+
+### Other
+
+
 ## Release 1.3.0 (2021-10-23)
 
 **rliable plots and bug fixes**
diff --git a/README.md b/README.md
index 072d46660..b0756643f 100644
--- a/README.md
+++ b/README.md
@@ -82,6 +82,7 @@ python scripts/all_plots.py -a sac td3 tqc --env HalfCheetah Ant -f rl-trained-a
 ## Plot with the rliable library
 
 The RL zoo integrates some of [rliable](https://agarwl.github.io/rliable/) library features.
+You can find a visual explanation of the tools used by rliable in this [blog post](https://araffin.github.io/post/rliable/).
 
 First, you need to install [rliable](https://github.com/google-research/rliable).
 
@@ -343,7 +344,7 @@ Additional Atari Games (to be completed):
 ### PyBullet Environments
 
 See https://github.com/bulletphysics/bullet3/tree/master/examples/pybullet/gym/pybullet_envs.
-Similar to [MuJoCo Envs](https://gym.openai.com/envs/#mujoco) but with a free simulator: pybullet. We are using `BulletEnv-v0` version.
+Similar to [MuJoCo Envs](https://gym.openai.com/envs/#mujoco) but with a ~free~ (MuJoCo 2.1.0+ is now free!) easy to install simulator: pybullet. We are using `BulletEnv-v0` version.
 
 Note: those environments are derived from [Roboschool](https://github.com/openai/roboschool) and are harder than the Mujoco version (see [Pybullet issue](https://github.com/bulletphysics/bullet3/issues/1718#issuecomment-393198883))
 
@@ -367,6 +368,16 @@ PyBullet Envs (Continued)
 | TD3      | | | | |
 | TQC      | | | | |
 
+### MuJoCo Environments
+
+|  RL Algo |  Walker2d | HalfCheetah | Ant | Swimmer |  Hopper | Humanoid |
+|----------|-----------|-------------|-----|---------|---------|----------|
+| A2C      |  | :heavy_check_mark: |  | :heavy_check_mark: | :heavy_check_mark: | |
+| PPO      | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | |
+| DDPG     |  |  |  |  |  | |
+| SAC      | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |
+| TD3      | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |
+| TQC      | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |
 
 ### Robotics Environments
 
@@ -386,7 +397,7 @@ We used the v1 environments.
 
 See https://github.com/qgallouedec/panda-gym/.
 
-Similar to [MuJoCo Robotics Envs](https://gym.openai.com/envs/#robotics) but with a free simulator: pybullet.
+Similar to [MuJoCo Robotics Envs](https://gym.openai.com/envs/#robotics) but with a ~free~ easy to install simulator: pybullet.
 
 We used the v1 environments.
 
diff --git a/benchmark.md b/benchmark.md
index 8d38d915d..e37e8b000 100644
--- a/benchmark.md
+++ b/benchmark.md
@@ -26,7 +26,9 @@ and also allow users to have access to pretrained agents.*
 |a2c  |BreakoutNoFrameskip-v4     |    279.793|   122.177|10M        |        604115|           82|
 |a2c  |CartPole-v1                |    500.000|     0.000|500k       |        150000|          300|
 |a2c  |EnduroNoFrameskip-v4       |      0.000|     0.000|10M        |        599040|           45|
+|a2c  |HalfCheetah-v3             |   3041.174|   157.265|1M         |        150000|          150|
 |a2c  |HalfCheetahBulletEnv-v0    |   2107.384|    36.008|2M         |        150000|          150|
+|a2c  |Hopper-v3                  |    733.454|   376.574|1M         |        149987|          580|
 |a2c  |HopperBulletEnv-v0         |    815.355|   313.798|2M         |        149541|          254|
 |a2c  |LunarLander-v2             |    155.751|    80.419|200k       |        149443|          297|
 |a2c  |LunarLanderContinuous-v2   |     84.225|   145.906|5M         |        149305|          256|
@@ -39,6 +41,7 @@ and also allow users to have access to pretrained agents.*
 |a2c  |RoadRunnerNoFrameskip-v4   |  31671.512|  6364.085|10M        |        606710|          172|
 |a2c  |SeaquestNoFrameskip-v4     |   1721.493|   105.339|10M        |        599691|           67|
 |a2c  |SpaceInvadersNoFrameskip-v4|    627.160|   201.974|10M        |        604848|          162|
+|a2c  |Swimmer-v3                 |    200.627|     2.544|1M         |        150000|          150|
 |a2c  |Walker2DBulletEnv-v0       |    858.209|   333.116|2M         |        149156|          173|
 |ddpg |AntBulletEnv-v0            |   2399.147|    75.410|1M         |        150000|          150|
 |ddpg |BipedalWalker-v3           |    197.486|   141.580|1M         |        149237|          227|
@@ -63,6 +66,7 @@ and also allow users to have access to pretrained agents.*
 |dqn  |SeaquestNoFrameskip-v4     |   2000.290|   606.644|10M        |        599505|           69|
 |dqn  |SpaceInvadersNoFrameskip-v4|    622.742|   201.564|10M        |        604311|          155|
 |ppo  |Acrobot-v1                 |    -73.506|    18.201|1M         |        149979|         2013|
+|ppo  |Ant-v3                     |   1327.158|   451.577|1M         |        149572|          175|
 |ppo  |AntBulletEnv-v0            |   2865.922|    56.468|2M         |        150000|          150|
 |ppo  |AsteroidsNoFrameskip-v4    |   2156.174|   744.640|10M        |        602092|          149|
 |ppo  |BeamRiderNoFrameskip-v4    |   3397.000|  1662.368|10M        |        598926|           46|
@@ -71,7 +75,9 @@ and also allow users to have access to pretrained agents.*
 |ppo  |BreakoutNoFrameskip-v4     |    398.033|    33.328|10M        |        600418|           60|
 |ppo  |CartPole-v1                |    500.000|     0.000|100k       |        150000|          300|
 |ppo  |EnduroNoFrameskip-v4       |    996.364|   176.090|10M        |        572416|           11|
+|ppo  |HalfCheetah-v3             |   5819.099|   663.530|1M         |        150000|          150|
 |ppo  |HalfCheetahBulletEnv-v0    |   2924.721|    64.465|2M         |        150000|          150|
+|ppo  |Hopper-v3                  |   2410.435|    10.026|1M         |        150000|          150|
 |ppo  |HopperBulletEnv-v0         |   2575.054|   223.301|2M         |        149094|          152|
 |ppo  |LunarLander-v2             |    242.119|    31.823|1M         |        149636|          369|
 |ppo  |LunarLanderContinuous-v2   |    270.863|    32.072|1M         |        149956|          526|
@@ -84,7 +90,9 @@ and also allow users to have access to pretrained agents.*
 |ppo  |RoadRunnerNoFrameskip-v4   |  40680.645|  6675.058|10M        |        605786|          155|
 |ppo  |SeaquestNoFrameskip-v4     |   1783.636|    34.096|10M        |        598243|           66|
 |ppo  |SpaceInvadersNoFrameskip-v4|    960.331|   425.355|10M        |        603771|          136|
+|ppo  |Swimmer-v3                 |    281.561|     9.671|1M         |        150000|          150|
 |ppo  |Walker2DBulletEnv-v0       |   2109.992|    13.899|2M         |        150000|          150|
+|ppo  |Walker2d-v3                |   3478.798|   821.708|1M         |        149343|          171|
 |qrdqn|Acrobot-v1                 |    -69.135|     9.967|100k       |        149949|         2138|
 |qrdqn|AsteroidsNoFrameskip-v4    |   2185.303|  1097.172|10M        |        599784|           66|
 |qrdqn|BeamRiderNoFrameskip-v4    |  17122.941| 10769.997|10M        |        596483|           17|
@@ -98,26 +106,39 @@ and also allow users to have access to pretrained agents.*
 |qrdqn|RoadRunnerNoFrameskip-v4   |  42325.424|  8361.161|10M        |        591016|           59|
 |qrdqn|SeaquestNoFrameskip-v4     |   2557.576|    76.951|10M        |        596275|           66|
 |qrdqn|SpaceInvadersNoFrameskip-v4|   1899.928|   823.488|10M        |        597218|           69|
+|sac  |Ant-v3                     |   4615.791|  1354.111|1M         |        149074|          165|
 |sac  |AntBulletEnv-v0            |   3073.114|   175.148|1M         |        150000|          150|
 |sac  |BipedalWalker-v3           |    297.668|    33.060|500k       |        149530|          136|
 |sac  |BipedalWalkerHardcore-v3   |      4.423|   103.910|10M        |        149794|           88|
+|sac  |HalfCheetah-v3             |   9535.451|   100.470|1M         |        150000|          150|
 |sac  |HalfCheetahBulletEnv-v0    |   2792.170|    12.088|1M         |        150000|          150|
+|sac  |Hopper-v3                  |   2325.547|  1129.676|1M         |        149841|          236|
 |sac  |HopperBulletEnv-v0         |   2603.494|   164.322|1M         |        149724|          151|
+|sac  |Humanoid-v3                |   6232.287|   279.885|2M         |        149460|          150|
 |sac  |LunarLanderContinuous-v2   |    260.390|    65.467|500k       |        149634|          672|
 |sac  |MountainCarContinuous-v0   |     94.679|     1.134|50k        |        149966|         1443|
 |sac  |Pendulum-v0                |   -156.995|    88.714|20k        |        150000|          750|
 |sac  |ReacherBulletEnv-v0        |     18.062|     9.729|300k       |        150000|         1000|
+|sac  |Swimmer-v3                 |    345.568|     3.084|1M         |        150000|          150|
 |sac  |Walker2DBulletEnv-v0       |   2292.266|    13.970|1M         |        149983|          150|
+|sac  |Walker2d-v3                |   3863.203|   254.347|1M         |        149309|          150|
+|td3  |Ant-v3                     |   5813.274|   589.773|1M         |        149393|          151|
 |td3  |AntBulletEnv-v0            |   3300.026|    54.640|1M         |        150000|          150|
 |td3  |BipedalWalker-v3           |    305.990|    56.886|1M         |        149999|          224|
 |td3  |BipedalWalkerHardcore-v3   |    -98.116|    16.087|10M        |        150000|           75|
+|td3  |HalfCheetah-v3             |   9655.666|   969.916|1M         |        150000|          150|
 |td3  |HalfCheetahBulletEnv-v0    |   2821.641|    19.722|1M         |        150000|          150|
+|td3  |Hopper-v3                  |   3606.390|     4.027|1M         |        150000|          150|
 |td3  |HopperBulletEnv-v0         |   2681.609|    27.806|1M         |        149486|          150|
+|td3  |Humanoid-v3                |   5566.687|    14.544|2M         |        150000|          150|
 |td3  |LunarLanderContinuous-v2   |    207.451|    67.562|300k       |        149488|          337|
 |td3  |MountainCarContinuous-v0   |     93.483|     0.075|300k       |        149976|         2275|
 |td3  |Pendulum-v0                |   -151.855|    90.227|20k        |        150000|          750|
 |td3  |ReacherBulletEnv-v0        |     17.114|     9.750|300k       |        150000|         1000|
+|td3  |Swimmer-v3                 |    359.127|     1.244|1M         |        150000|          150|
 |td3  |Walker2DBulletEnv-v0       |   2213.672|   230.558|1M         |        149800|          152|
+|td3  |Walker2d-v3                |   4717.823|    46.303|1M         |        150000|          150|
+|tqc  |Ant-v3                     |   3339.362|  1969.906|1M         |        149583|          202|
 |tqc  |AntBulletEnv-v0            |   3456.717|   248.733|1M         |        150000|          150|
 |tqc  |BipedalWalker-v3           |    329.808|    45.083|500k       |        149682|          254|
 |tqc  |BipedalWalkerHardcore-v3   |    235.226|   110.569|2M         |        149032|          131|
@@ -125,8 +146,11 @@ and also allow users to have access to pretrained agents.*
 |tqc  |FetchPush-v1               |     -8.799|     5.438|1M         |        150000|         3000|
 |tqc  |FetchReach-v1              |     -1.659|     0.873|20k        |        150000|         3000|
 |tqc  |FetchSlide-v1              |    -29.210|    11.387|3M         |        150000|         3000|
+|tqc  |HalfCheetah-v3             |  12089.939|   127.440|1M         |        150000|          150|
 |tqc  |HalfCheetahBulletEnv-v0    |   3675.299|    17.681|1M         |        150000|          150|
+|tqc  |Hopper-v3                  |   3754.199|     8.276|1M         |        150000|          150|
 |tqc  |HopperBulletEnv-v0         |   2662.373|   206.210|1M         |        149881|          151|
+|tqc  |Humanoid-v3                |   7239.320|  1647.498|2M         |        149508|          165|
 |tqc  |LunarLanderContinuous-v2   |    277.956|    25.466|500k       |        149928|          706|
 |tqc  |MountainCarContinuous-v0   |     63.641|    45.259|50k        |        149796|          186|
 |tqc  |PandaPickAndPlace-v1       |     -8.024|     6.674|1M         |        150000|         3000|
@@ -136,5 +160,7 @@ and also allow users to have access to pretrained agents.*
 |tqc  |PandaStack-v1              |    -96.915|    17.240|1M         |        150000|         1500|
 |tqc  |Pendulum-v0                |   -151.340|    87.893|20k        |        150000|          750|
 |tqc  |ReacherBulletEnv-v0        |     18.255|     9.543|300k       |        150000|         1000|
+|tqc  |Swimmer-v3                 |    339.423|     1.486|1M         |        150000|          150|
 |tqc  |Walker2DBulletEnv-v0       |   2508.934|   614.624|1M         |        149572|          159|
+|tqc  |Walker2d-v3                |   4380.720|   500.489|1M         |        149606|          152|
 |tqc  |parking-v0                 |     -6.762|     2.690|100k       |        149983|         7528|
diff --git a/hyperparams/a2c.yml b/hyperparams/a2c.yml
index cc1e4e7af..9f8dc3887 100644
--- a/hyperparams/a2c.yml
+++ b/hyperparams/a2c.yml
@@ -159,3 +159,27 @@ HopperBulletEnv-v0:
 ReacherBulletEnv-v0:
   <<: *pybullet-defaults
   learning_rate: lin_0.0008
+
+# === Mujoco Envs ===
+
+HalfCheetah-v3: &mujoco-defaults
+  normalize: true
+  n_timesteps: !!float 1e6
+  policy: 'MlpPolicy'
+
+Ant-v3:
+  <<: *mujoco-defaults
+
+Hopper-v3:
+  <<: *mujoco-defaults
+
+Walker2d-v3:
+  <<: *mujoco-defaults
+
+Humanoid-v3:
+  <<: *mujoco-defaults
+  n_timesteps: !!float 2e6
+
+Swimmer-v3:
+  <<: *mujoco-defaults
+  gamma: 0.9999
diff --git a/hyperparams/ddpg.yml b/hyperparams/ddpg.yml
index 13a9b9ede..e86be61db 100644
--- a/hyperparams/ddpg.yml
+++ b/hyperparams/ddpg.yml
@@ -121,3 +121,31 @@ InvertedDoublePendulumBulletEnv-v0:
 InvertedPendulumSwingupBulletEnv-v0:
   <<: *pybullet-defaults
   n_timesteps: !!float 3e5
+
+# === Mujoco Envs ===
+
+HalfCheetah-v3: &mujoco-defaults
+  n_timesteps: !!float 1e6
+  policy: 'MlpPolicy'
+  learning_starts: 10000
+  noise_type: 'normal'
+  noise_std: 0.1
+
+Ant-v3:
+  <<: *mujoco-defaults
+
+Hopper-v3:
+  <<: *mujoco-defaults
+
+Walker2d-v3:
+  <<: *mujoco-defaults
+
+Humanoid-v3:
+  <<: *mujoco-defaults
+  n_timesteps: !!float 2e6
+
+Swimmer-v3:
+  <<: *mujoco-defaults
+  gamma: 0.9999
+  train_freq: 1
+  gradient_steps: 1
diff --git a/hyperparams/ppo.yml b/hyperparams/ppo.yml
index 1905e077b..6bf6428db 100644
--- a/hyperparams/ppo.yml
+++ b/hyperparams/ppo.yml
@@ -346,24 +346,37 @@ CarRacing-v0:
                        ortho_init=False,
                        )"
 
+
+# === Mujoco Envs ===
+# HalfCheetah-v3: &mujoco-defaults
+#   normalize: true
+#   n_timesteps: !!float 1e6
+#   policy: 'MlpPolicy'
+
+Ant-v3: &mujoco-defaults
+  normalize: true
+  n_timesteps: !!float 1e6
+  policy: 'MlpPolicy'
+
+# Hopper-v3:
+#   <<: *mujoco-defaults
+#
+# Walker2d-v3:
+#   <<: *mujoco-defaults
+#
+# Humanoid-v3:
+#   <<: *mujoco-defaults
+#   n_timesteps: !!float 2e6
+#
+Swimmer-v3:
+  <<: *mujoco-defaults
+  gamma: 0.9999
+
 # Tuned
 # 10 mujoco envs
-Ant-v2:
-  n_envs: 1
-  policy: 'MlpPolicy'
-  n_timesteps: !!float 1e7
-  batch_size: 32
-  n_steps: 512
-  gamma: 0.98
-  learning_rate: 1.90609e-05
-  ent_coef: 4.9646e-07
-  clip_range: 0.1
-  n_epochs: 10
-  gae_lambda: 0.8
-  max_grad_norm: 0.6
-  vf_coef: 0.677239
 
-HalfCheetah-v2:
+HalfCheetah-v3:
+  normalize: true
   n_envs: 1
   policy: 'MlpPolicy'
   n_timesteps: !!float 1e6
@@ -384,7 +397,24 @@ HalfCheetah-v2:
                     net_arch=[dict(pi=[256, 256], vf=[256, 256])]
                   )"
 
-Hopper-v2:
+# Ant-v3:
+#   normalize: true
+#   n_envs: 1
+#   policy: 'MlpPolicy'
+#   n_timesteps: !!float 1e7
+#   batch_size: 32
+#   n_steps: 512
+#   gamma: 0.98
+#   learning_rate: 1.90609e-05
+#   ent_coef: 4.9646e-07
+#   clip_range: 0.1
+#   n_epochs: 10
+#   gae_lambda: 0.8
+#   max_grad_norm: 0.6
+#   vf_coef: 0.677239
+
+Hopper-v3:
+  normalize: true
   n_envs: 1
   policy: 'MlpPolicy'
   n_timesteps: !!float 1e6
@@ -405,7 +435,8 @@ Hopper-v2:
                     net_arch=[dict(pi=[256, 256], vf=[256, 256])]
                   )"
 
-HumanoidStandup-v2:
+HumanoidStandup-v3:
+  normalize: true
   n_envs: 1
   policy: 'MlpPolicy'
   n_timesteps: !!float 1e7
@@ -426,7 +457,8 @@ HumanoidStandup-v2:
                     net_arch=[dict(pi=[256, 256], vf=[256, 256])]
                   )"
 
-Humanoid-v2:
+Humanoid-v3:
+  normalize: true
   n_envs: 1
   policy: 'MlpPolicy'
   n_timesteps: !!float 1e7
@@ -447,7 +479,8 @@ Humanoid-v2:
                     net_arch=[dict(pi=[256, 256], vf=[256, 256])]
                   )"
 
-InvertedDoublePendulum-v2:
+InvertedDoublePendulum-v3:
+  normalize: true
   n_envs: 1
   policy: 'MlpPolicy'
   n_timesteps: !!float 1e6
@@ -462,7 +495,8 @@ InvertedDoublePendulum-v2:
   max_grad_norm: 0.5
   vf_coef: 0.695929
 
-InvertedPendulum-v2:
+InvertedPendulum-v3:
+  normalize: true
   n_envs: 1
   policy: 'MlpPolicy'
   n_timesteps: !!float 1e6
@@ -477,7 +511,8 @@ InvertedPendulum-v2:
   max_grad_norm: 0.3
   vf_coef: 0.19816
 
-Reacher-v2:
+Reacher-v3:
+  normalize: true
   n_envs: 1
   policy: 'MlpPolicy'
   n_timesteps: !!float 1e6
@@ -492,28 +527,30 @@ Reacher-v2:
   max_grad_norm: 0.9
   vf_coef: 0.950368
 
-Swimmer-v2:
-  n_envs: 1
-  policy: 'MlpPolicy'
-  n_timesteps: !!float 1e6
-  batch_size: 32
-  n_steps: 512
-  gamma: 0.9999
-  learning_rate: 5.49717e-05
-  ent_coef: 0.0554757
-  clip_range: 0.3
-  n_epochs: 10
-  gae_lambda: 0.95
-  max_grad_norm: 0.6
-  vf_coef: 0.38782
-  policy_kwargs: "dict(
-                    log_std_init=-2,
-                    ortho_init=False,
-                    activation_fn=nn.ReLU,
-                    net_arch=[dict(pi=[256, 256], vf=[256, 256])]
-                  )"
-
-Walker2d-v2:
+# Swimmer-v3:
+#   normalize: true
+#   n_envs: 1
+#   policy: 'MlpPolicy'
+#   n_timesteps: !!float 1e6
+#   batch_size: 32
+#   n_steps: 512
+#   gamma: 0.9999
+#   learning_rate: 5.49717e-05
+#   ent_coef: 0.0554757
+#   clip_range: 0.3
+#   n_epochs: 10
+#   gae_lambda: 0.95
+#   max_grad_norm: 0.6
+#   vf_coef: 0.38782
+#   policy_kwargs: "dict(
+#                     log_std_init=-2,
+#                     ortho_init=False,
+#                     activation_fn=nn.ReLU,
+#                     net_arch=[dict(pi=[256, 256], vf=[256, 256])]
+#                   )"
+
+Walker2d-v3:
+  normalize: true
   n_envs: 1
   policy: 'MlpPolicy'
   n_timesteps: !!float 1e6
diff --git a/hyperparams/sac.yml b/hyperparams/sac.yml
index c78eabfde..43178725b 100644
--- a/hyperparams/sac.yml
+++ b/hyperparams/sac.yml
@@ -189,6 +189,30 @@ CarRacing-v0:
   use_sde_at_warmup: True
   policy_kwargs: "dict(log_std_init=-2, net_arch=[64, 64])"
 
+# === Mujoco Envs ===
+
+HalfCheetah-v3: &mujoco-defaults
+  n_timesteps: !!float 1e6
+  policy: 'MlpPolicy'
+  learning_starts: 10000
+
+Ant-v3:
+  <<: *mujoco-defaults
+
+Hopper-v3:
+  <<: *mujoco-defaults
+
+Walker2d-v3:
+  <<: *mujoco-defaults
+
+Humanoid-v3:
+  <<: *mujoco-defaults
+  n_timesteps: !!float 2e6
+
+Swimmer-v3:
+  <<: *mujoco-defaults
+  gamma: 0.9999
+
 # === HER Robotics GoalEnvs ===
 
 FetchReach-v1:
diff --git a/hyperparams/td3.yml b/hyperparams/td3.yml
index 83ed472a5..e761aab42 100644
--- a/hyperparams/td3.yml
+++ b/hyperparams/td3.yml
@@ -123,3 +123,42 @@ MinitaurBulletEnv-v0:
   train_freq: 1
   gradient_steps: 1
   policy_kwargs: "dict(net_arch=[400, 300])"
+
+# === Mujoco Envs ===
+
+HalfCheetah-v3: &mujoco-defaults
+  n_timesteps: !!float 1e6
+  policy: 'MlpPolicy'
+  learning_starts: 10000
+  noise_type: 'normal'
+  noise_std: 0.1
+
+Ant-v3:
+  <<: *mujoco-defaults
+
+Hopper-v3:
+  <<: *mujoco-defaults
+  # SAC Hyperparams
+  train_freq: 1
+  gradient_steps: 1
+  learning_rate: !!float 3e-4
+  batch_size: 256
+
+Walker2d-v3:
+  <<: *mujoco-defaults
+
+Humanoid-v3:
+  <<: *mujoco-defaults
+  n_timesteps: !!float 2e6
+  # SAC Hyperparams
+  train_freq: 1
+  gradient_steps: 1
+  learning_rate: !!float 3e-4
+  batch_size: 256
+
+# Tuned
+Swimmer-v3:
+  <<: *mujoco-defaults
+  gamma: 0.9999
+  train_freq: 1
+  gradient_steps: 1
diff --git a/hyperparams/tqc.yml b/hyperparams/tqc.yml
index 53825f50e..f10dc4aff 100644
--- a/hyperparams/tqc.yml
+++ b/hyperparams/tqc.yml
@@ -138,6 +138,30 @@ MinitaurBulletEnv-v0:
   learning_starts: 10000
   # top_quantiles_to_drop_per_net: 5
 
+# === Mujoco Envs ===
+
+HalfCheetah-v3: &mujoco-defaults
+  n_timesteps: !!float 1e6
+  policy: 'MlpPolicy'
+  learning_starts: 10000
+
+Ant-v3:
+  <<: *mujoco-defaults
+
+Hopper-v3:
+  <<: *mujoco-defaults
+  top_quantiles_to_drop_per_net: 5
+
+Walker2d-v3:
+  <<: *mujoco-defaults
+
+Humanoid-v3:
+  <<: *mujoco-defaults
+  n_timesteps: !!float 2e6
+
+Swimmer-v3:
+  <<: *mujoco-defaults
+  gamma: 0.9999
 
 # === HER Robotics GoalEnvs ===
 FetchReach-v1:
diff --git a/logs/benchmark/a2c-HalfCheetah-v3/0.monitor.csv b/logs/benchmark/a2c-HalfCheetah-v3/0.monitor.csv
new file mode 100644
index 000000000..523b3cf3b
--- /dev/null
+++ b/logs/benchmark/a2c-HalfCheetah-v3/0.monitor.csv
@@ -0,0 +1,152 @@
+#{"t_start": 1637100393.0826447, "env_id": "HalfCheetah-v3"}
+r,l,t
+3113.328499,1000,4.245912
+3169.092553,1000,5.742782
+2935.482096,1000,7.242378
+3051.672893,1000,8.742238
+3240.469021,1000,10.239377
+3075.263276,1000,11.732651
+3173.476671,1000,13.221114
+3060.425292,1000,14.721946
+3023.028443,1000,16.213673
+3123.833214,1000,17.712484
+3295.370529,1000,19.198954
+3237.089278,1000,20.693489
+3202.709894,1000,22.1866
+2778.819387,1000,23.686388
+3184.33042,1000,25.184407
+3134.025481,1000,26.675004
+2933.729871,1000,28.164711
+2733.55796,1000,29.667303
+2945.604338,1000,31.163986
+3004.15973,1000,32.660174
+3076.608986,1000,34.162555
+3258.233072,1000,35.662504
+3091.832836,1000,37.156379
+3008.270395,1000,38.649915
+2787.002096,1000,40.135433
+2955.259192,1000,41.624981
+2974.273025,1000,43.113276
+3266.940174,1000,44.603306
+2749.212787,1000,46.108447
+3005.820941,1000,47.600123
+3120.293044,1000,49.085116
+3088.711462,1000,50.585363
+3004.742595,1000,52.07283
+2991.252075,1000,53.581133
+3084.238891,1000,55.079151
+3104.522424,1000,56.561399
+2989.027671,1000,58.051384
+3028.461421,1000,59.537698
+3284.00287,1000,61.025674
+3157.424394,1000,62.523677
+3074.405108,1000,64.013909
+3138.51929,1000,65.507016
+3096.315885,1000,66.997354
+2990.18761,1000,68.469725
+2861.590867,1000,69.723632
+3206.200818,1000,70.982261
+3256.41213,1000,72.239456
+3136.426875,1000,73.554631
+2854.265591,1000,75.04669
+3041.405102,1000,76.537343
+2752.371908,1000,78.027554
+3068.585393,1000,79.320504
+2993.955951,1000,80.616149
+3062.028316,1000,82.115277
+3038.400627,1000,83.61012
+2988.195501,1000,85.107924
+3163.919608,1000,86.595743
+3059.575055,1000,88.083019
+3193.233333,1000,89.57748
+2639.345387,1000,91.069109
+2918.025266,1000,92.552843
+2893.167672,1000,94.054347
+2850.365982,1000,95.549581
+3227.902419,1000,97.037523
+3410.645012,1000,98.53885
+3081.574435,1000,100.046111
+2911.786281,1000,101.537082
+2840.094733,1000,103.037886
+2904.399402,1000,104.525537
+3224.980785,1000,106.023033
+2798.045846,1000,107.52136
+3284.819612,1000,109.034362
+3181.292083,1000,110.542263
+3048.037867,1000,112.03875
+2695.733256,1000,113.536082
+2944.309285,1000,115.028183
+3126.777844,1000,116.522027
+3066.535502,1000,118.015832
+3127.785459,1000,119.513995
+3152.354853,1000,121.00385
+3200.712239,1000,122.497931
+3158.451285,1000,123.985014
+2887.013675,1000,125.482738
+2977.472984,1000,126.977459
+2930.057442,1000,128.467474
+3318.369489,1000,129.957006
+2887.45445,1000,131.45058
+2956.592672,1000,132.947507
+3042.691247,1000,134.441623
+2980.294842,1000,135.93718
+2969.331011,1000,137.426772
+3235.959877,1000,138.938947
+3074.039576,1000,140.434766
+3386.910209,1000,141.928782
+2671.002272,1000,143.42679
+3098.23398,1000,144.919593
+3111.543596,1000,146.415758
+3233.896466,1000,147.918278
+2790.431174,1000,149.410203
+3198.153114,1000,150.906173
+3112.200964,1000,152.398817
+2942.846021,1000,153.888179
+3132.329097,1000,155.384869
+3062.637064,1000,156.887796
+3101.645086,1000,158.373999
+2916.281255,1000,159.866917
+2891.716768,1000,161.365925
+2972.848377,1000,162.862531
+3188.344648,1000,164.35216
+3111.715903,1000,165.843313
+3185.577622,1000,167.33691
+3034.545425,1000,168.827543
+3332.356394,1000,170.321867
+3194.2285,1000,171.791021
+3003.877708,1000,173.279531
+3329.464335,1000,174.765413
+2888.483037,1000,176.257762
+2901.476549,1000,177.744766
+3035.328399,1000,179.102669
+3207.122762,1000,180.285807
+2569.055332,1000,181.476305
+3122.287915,1000,182.658184
+3009.605189,1000,183.836717
+3009.413487,1000,185.024866
+3162.58435,1000,186.393503
+2926.936303,1000,187.88523
+3009.066134,1000,189.385158
+3050.270804,1000,190.872826
+2774.306592,1000,192.366162
+3053.062059,1000,193.875107
+2959.055902,1000,195.374697
+3142.138939,1000,196.862913
+3125.467842,1000,198.359393
+3107.625865,1000,199.860254
+2741.578917,1000,201.355576
+3118.712699,1000,202.851107
+2944.780222,1000,204.341509
+3013.329179,1000,205.836141
+2883.918787,1000,207.324185
+2641.898309,1000,208.812242
+3130.311972,1000,210.31072
+2976.444288,1000,211.791868
+3166.233353,1000,213.291624
+3004.723508,1000,214.796363
+3167.319527,1000,216.288506
+3034.841617,1000,217.771178
+2805.405301,1000,219.265632
+2942.793681,1000,220.755541
+2967.422693,1000,222.234992
+3216.29069,1000,223.725725
diff --git a/logs/benchmark/a2c-Hopper-v3/0.monitor.csv b/logs/benchmark/a2c-Hopper-v3/0.monitor.csv
new file mode 100644
index 000000000..5bf5168a5
--- /dev/null
+++ b/logs/benchmark/a2c-Hopper-v3/0.monitor.csv
@@ -0,0 +1,582 @@
+#{"t_start": 1637100844.0645404, "env_id": "Hopper-v3"}
+r,l,t
+704.284499,264,3.12659
+834.80132,277,3.480355
+912.374845,291,3.851754
+209.4828,123,4.007211
+1067.924902,353,4.477641
+895.179755,284,4.913578
+878.333262,285,5.345465
+245.339242,132,5.547758
+786.919779,267,5.953008
+255.440718,133,6.15685
+1091.816766,353,6.69338
+1112.147615,363,7.249534
+911.025368,316,7.730974
+242.344458,131,7.931059
+268.919973,132,8.129461
+1040.448012,344,8.657079
+216.106438,126,8.851108
+730.073084,262,9.253112
+1234.906533,417,9.893604
+811.432339,271,10.308408
+857.388908,289,10.752788
+271.348125,131,10.951149
+1036.336269,341,11.473908
+1200.678079,391,12.073875
+250.461407,130,12.273641
+865.18278,286,12.713169
+1075.951406,349,13.249269
+245.453873,136,13.459762
+742.208885,234,13.818134
+254.678026,135,14.02705
+885.416017,296,14.482904
+1332.438181,442,15.160303
+263.753842,135,15.368726
+1094.850656,370,15.934918
+279.626045,137,16.140245
+226.591873,129,16.337291
+1092.451221,361,16.886536
+1087.024221,362,17.432871
+798.476461,275,17.852629
+881.407173,276,18.271237
+816.374265,267,18.677103
+287.491165,138,18.88866
+1009.468174,310,19.360788
+738.673729,264,19.764548
+925.934384,294,20.213972
+1200.22404,399,20.823342
+1105.516132,361,21.378353
+1522.434183,496,22.136836
+1043.42476,335,22.652526
+278.34245,137,22.863543
+276.840301,134,23.069773
+1061.812597,345,23.603499
+264.901483,135,23.807805
+260.131947,136,24.016535
+927.736664,290,24.459324
+985.656798,324,24.95362
+1092.799869,356,25.500727
+997.289762,305,25.966257
+876.525869,288,26.405125
+257.534559,129,26.601996
+263.719942,136,26.81115
+806.857096,269,27.222071
+1252.707094,412,27.852499
+819.82385,292,28.301388
+692.821154,267,28.70997
+788.460333,238,29.075353
+1070.876284,332,29.587313
+266.277018,132,29.792308
+290.353909,138,30.007074
+261.796797,136,30.216615
+1045.815746,311,30.692615
+271.318902,132,30.894464
+1092.440054,352,31.435695
+1024.511291,333,31.947435
+733.715139,225,32.291796
+1337.217409,436,32.963427
+1071.943419,349,33.503492
+1043.734569,344,34.033379
+826.09393,274,34.452631
+285.200587,137,34.664612
+262.497352,133,34.865554
+1011.694903,342,35.390616
+274.597016,137,35.601987
+257.441415,133,35.807999
+1012.060651,343,36.331584
+249.09608,133,36.536805
+255.245733,135,36.744347
+271.577267,131,36.943876
+258.908622,130,37.143466
+1050.068938,330,37.654881
+268.446429,134,37.858187
+260.651845,132,38.059581
+278.139316,134,38.266948
+1010.018972,337,38.783065
+1058.804175,354,39.322935
+1253.031754,410,39.951973
+226.620405,124,40.14445
+1053.21752,344,40.67774
+257.843853,134,40.882101
+1045.970271,330,41.386255
+998.836405,331,41.896541
+1298.303674,434,42.561584
+274.097205,133,42.763661
+228.374072,127,42.96048
+801.798258,307,43.428912
+1425.524515,489,44.18192
+1015.366436,331,44.687972
+772.273945,259,45.086257
+690.960031,253,45.473974
+1239.02355,423,46.119266
+919.017087,321,46.604733
+826.304263,275,46.963493
+1366.454742,434,47.538224
+254.163686,127,47.704572
+245.124825,130,47.874267
+244.175416,127,48.041099
+238.342549,129,48.212877
+1356.935864,451,48.808701
+1142.429136,366,49.28844
+962.7549,327,49.715348
+977.27529,312,50.123616
+828.33946,281,50.494879
+284.048743,135,50.673868
+943.359476,300,51.069202
+233.127775,126,51.234977
+1040.32163,327,51.666471
+255.334353,127,51.834708
+1071.608232,343,52.285825
+1079.553242,361,52.767215
+846.66045,299,53.167396
+255.249643,136,53.345463
+819.214994,279,53.713834
+787.640225,270,54.071084
+1132.182573,365,54.553128
+1140.022444,374,55.046108
+887.99287,325,55.477224
+921.497124,295,55.867113
+255.386798,131,56.040387
+280.448583,134,56.214861
+1138.400715,372,56.715127
+252.501156,129,56.917694
+810.097297,270,57.334878
+920.069791,313,57.814457
+1109.588389,353,58.354921
+826.023163,267,58.767101
+1223.490819,382,59.356444
+885.995651,291,59.801033
+1026.510961,338,60.320965
+716.027516,259,60.719942
+961.043216,308,61.192756
+879.570051,289,61.642851
+987.307505,310,62.128357
+1164.045496,370,62.692744
+1029.709011,336,63.210014
+1452.843403,513,64.003625
+1324.997338,427,64.659564
+825.202197,277,65.09002
+264.489156,139,65.304841
+1030.02556,344,65.832852
+990.599436,320,66.325145
+980.666016,320,66.81463
+252.42755,135,67.022357
+835.776985,276,67.446821
+230.833623,127,67.640417
+850.46731,266,68.046959
+715.523727,244,68.422432
+1065.240399,354,68.964422
+236.643755,129,69.161491
+275.308061,134,69.367726
+524.668063,173,69.631363
+239.28097,134,69.837732
+223.548762,129,70.036969
+908.135022,295,70.489339
+214.05955,125,70.680558
+1124.154284,408,71.304928
+280.299081,135,71.512342
+268.794563,138,71.722419
+288.032291,138,71.933435
+1230.077756,404,72.557652
+1129.179053,389,73.158524
+265.880508,130,73.358871
+966.181812,296,73.812583
+1073.352462,357,74.36225
+784.999927,248,74.742617
+811.343063,274,75.166112
+237.950528,130,75.365282
+631.936719,197,75.668733
+1092.844721,368,76.235828
+828.784977,277,76.666007
+687.223925,213,76.990773
+216.832476,125,77.18214
+280.632772,134,77.390896
+746.153114,277,77.811788
+1349.758626,429,78.462934
+918.360003,328,78.972498
+737.09237,228,79.326261
+279.12194,133,79.532717
+235.390392,135,79.739088
+234.44227,132,79.940098
+1049.143508,346,80.470213
+1051.004892,339,80.995464
+266.682254,139,81.209114
+1068.955905,354,81.74953
+978.756884,312,82.233822
+226.476637,129,82.43106
+251.941452,132,82.63573
+1142.558406,358,83.187396
+815.523293,276,83.610011
+274.628322,134,83.813652
+250.863601,129,84.011891
+275.38031,136,84.219538
+801.359968,271,84.632561
+1339.493358,449,85.32478
+1370.717312,450,86.021789
+1046.422499,325,86.521462
+956.881389,306,86.99593
+276.340008,136,87.207437
+986.520016,341,87.732301
+932.602431,299,88.193356
+880.263197,295,88.644954
+954.570143,345,89.169847
+283.74898,135,89.377629
+247.037818,131,89.575146
+786.564153,263,89.980291
+1062.344529,344,90.507509
+1017.701792,319,90.997935
+1100.121145,354,91.541901
+962.868179,311,92.020341
+819.448652,271,92.439007
+1340.43005,434,93.104089
+1210.303966,413,93.736801
+1218.348164,396,94.343715
+739.990824,246,94.722957
+1093.907652,358,95.278782
+277.756673,134,95.484777
+912.371523,288,95.926319
+945.517986,315,96.371516
+890.058339,314,96.778258
+1292.843539,420,97.389334
+931.332262,306,97.856819
+955.132216,294,98.308453
+766.59574,249,98.688788
+806.281406,250,99.069505
+1092.801544,354,99.611418
+274.544924,136,99.818524
+264.356626,132,100.022401
+958.133398,328,100.526878
+242.476597,127,100.725023
+286.69142,136,100.929631
+845.397019,271,101.347212
+272.711219,134,101.556264
+259.758921,134,101.759782
+833.494166,259,102.157048
+930.044214,298,102.616075
+267.184298,133,102.820618
+1033.879953,312,103.295283
+273.041069,132,103.49741
+1303.425606,402,104.112136
+1107.972377,359,104.66397
+256.613903,133,104.865529
+1071.134506,349,105.401653
+1071.348083,370,105.965406
+880.957402,283,106.403691
+1033.160028,350,106.942961
+853.660436,280,107.37147
+265.736098,135,107.580718
+1030.401209,337,108.09876
+1228.293285,430,108.753512
+1053.650427,364,109.315005
+838.696053,291,109.762036
+954.13354,300,110.22315
+228.150138,128,110.421088
+293.339279,137,110.629073
+713.933281,264,111.033247
+943.353532,305,111.500482
+1341.162738,447,112.183795
+760.866858,255,112.572512
+246.016701,135,112.783088
+269.384349,135,112.98898
+243.12404,130,113.190116
+268.762041,136,113.399819
+1439.337056,457,114.111422
+786.453522,245,114.484675
+1069.826555,343,115.01698
+790.643949,274,115.437355
+801.759079,292,115.88569
+264.711654,130,116.085771
+246.399113,133,116.290355
+268.621655,132,116.490852
+713.741039,222,116.832213
+1242.676211,397,117.44217
+1313.423207,465,118.160904
+1222.72165,402,118.776468
+1053.783956,368,119.346657
+796.459454,264,119.751589
+1188.992156,386,120.344897
+197.19384,120,120.526337
+259.528668,137,120.734768
+1384.416611,450,121.429902
+864.53247,283,121.861681
+744.549858,235,122.220996
+832.396806,307,122.691467
+800.011397,288,123.13501
+288.636082,136,123.342932
+1016.876379,341,123.861523
+784.565795,261,124.263608
+913.441092,293,124.714219
+270.035176,130,124.910563
+939.917437,309,125.383921
+284.778542,134,125.590855
+1266.958182,412,126.220875
+286.068467,135,126.428392
+222.099235,126,126.619773
+744.683527,247,127.000829
+853.738421,281,127.433433
+826.902379,268,127.844858
+272.671022,136,128.054611
+814.396609,274,128.471751
+253.124521,133,128.678753
+1058.010381,346,129.205795
+827.6001,293,129.654249
+1224.286836,402,130.267304
+808.736923,272,130.681309
+808.285334,283,131.114218
+263.324384,139,131.329616
+953.230548,314,131.814003
+791.118312,261,132.212695
+277.654448,137,132.423259
+905.786708,293,132.873654
+250.231587,129,133.069796
+852.896009,279,133.489928
+1076.669861,341,134.007251
+894.233564,284,134.442857
+281.379484,135,134.650231
+813.248566,254,135.038337
+1043.319036,344,135.564043
+969.427408,317,136.046237
+232.797548,131,136.246332
+1306.972125,436,136.920708
+1000.910655,321,137.413944
+816.969749,289,137.859238
+865.551641,278,138.288634
+1348.283355,439,138.965055
+1122.805247,367,139.523389
+219.033272,124,139.711886
+1127.721848,357,140.253959
+1495.510219,482,140.995561
+1005.712205,311,141.474555
+693.664995,257,141.871892
+674.26996,256,142.268325
+864.633749,310,142.742644
+235.886135,130,142.943848
+267.66361,135,143.151784
+1094.855624,348,143.688267
+910.776712,300,144.146549
+699.619061,258,144.541098
+234.32039,126,144.737848
+236.74701,129,144.933879
+273.141631,137,145.14376
+881.287218,292,145.591296
+953.149078,297,146.044674
+266.66692,133,146.244951
+960.094965,318,146.734108
+282.784484,135,146.936796
+935.062191,281,147.370285
+1472.209167,476,148.101459
+291.836345,137,148.316572
+253.812383,127,148.507315
+262.191833,134,148.710787
+887.942076,271,149.121641
+276.301063,136,149.330479
+1334.621454,417,149.967785
+1137.776955,373,150.538931
+272.730849,135,150.746562
+965.410811,317,151.236633
+251.903795,129,151.42959
+238.03573,125,151.625107
+651.751242,201,151.934221
+242.521837,133,152.135551
+267.488826,135,152.339596
+922.405878,305,152.808814
+833.068362,279,153.234202
+276.731619,132,153.439548
+1015.043741,338,153.960827
+1028.318887,331,154.466741
+1201.609237,448,155.10748
+858.128768,312,155.518006
+237.598506,128,155.685002
+904.123917,294,156.07408
+278.141871,141,156.257742
+1222.356345,432,156.828303
+921.90217,324,157.254462
+292.155959,135,157.4347
+803.903485,279,157.805695
+1180.013031,405,158.336704
+1274.670271,429,158.900903
+948.707239,314,159.317069
+252.378307,133,159.493749
+971.709575,297,159.88517
+989.069617,328,160.323103
+1371.202939,448,160.9138
+779.26701,265,161.262568
+268.739256,131,161.432805
+266.670576,131,161.603922
+777.829495,263,161.947714
+955.665426,297,162.34181
+945.282127,308,162.752471
+723.089077,258,163.092363
+248.224194,128,163.260141
+1003.390018,313,163.673668
+1119.076314,372,164.166825
+249.727339,135,164.345854
+743.82807,248,164.674801
+1210.032337,395,165.190028
+244.959642,129,165.358963
+1279.334736,410,165.902628
+895.809104,287,166.279643
+245.847448,133,166.454262
+998.495196,324,166.875524
+975.779742,324,167.30229
+245.48777,133,167.479613
+696.421692,261,167.832534
+279.158361,137,168.046598
+230.041855,127,168.241601
+1455.803091,486,168.988898
+946.690002,336,169.506364
+851.950516,275,169.928308
+1018.898838,339,170.451134
+212.232833,122,170.640589
+1030.55296,337,171.159271
+273.593336,132,171.364572
+704.566654,217,171.695571
+937.280334,319,172.185896
+247.511496,134,172.393798
+266.212268,135,172.596332
+1110.057501,359,173.142248
+1190.93644,374,173.720017
+1096.214573,359,174.275208
+252.697214,131,174.478363
+220.435224,130,174.678283
+282.722864,137,174.890598
+916.972222,297,175.34481
+1285.797197,398,175.959135
+796.732349,269,176.375403
+1063.540768,359,176.929347
+277.028763,140,177.141622
+269.919481,137,177.353216
+261.493856,129,177.548123
+1223.223093,407,178.170331
+1337.458171,430,178.831802
+232.972832,131,179.034354
+265.56459,135,179.243788
+845.011968,273,179.665701
+818.285726,268,180.075065
+231.84282,127,180.268946
+1097.920062,375,180.844584
+292.96958,137,181.054963
+280.955355,134,181.259891
+1249.09038,388,181.864277
+839.801391,275,182.287453
+228.654461,126,182.478538
+235.27874,125,182.671323
+235.937595,129,182.865439
+252.650446,134,183.074562
+770.812062,263,183.478667
+264.843586,136,183.688614
+982.688691,333,184.20171
+260.132498,134,184.41264
+235.121679,128,184.607125
+247.915676,128,184.80404
+871.276773,286,185.241246
+1002.365274,346,185.776962
+234.163436,131,185.979201
+274.657548,132,186.182984
+1174.114603,383,186.774025
+930.583464,318,187.266435
+258.958174,132,187.470815
+1133.495961,381,188.05797
+251.89824,132,188.260767
+248.879692,133,188.447511
+822.985242,271,188.795381
+1424.684939,493,189.425983
+268.062453,135,189.619165
+1117.775401,367,190.181923
+1091.300157,352,190.719902
+270.197682,134,190.924951
+730.895954,262,191.326738
+966.519031,309,191.799062
+261.459691,137,192.008934
+239.512912,136,192.215852
+983.912915,340,192.735031
+948.171479,302,193.196705
+867.560877,280,193.626649
+826.862037,292,194.076579
+1074.790021,344,194.606823
+1089.305195,358,195.159873
+699.013397,218,195.49562
+1267.362079,432,196.158589
+271.056381,137,196.366058
+1089.381064,358,196.919215
+284.211843,134,197.123817
+240.567688,133,197.330941
+260.649018,134,197.537061
+1282.069063,439,198.202928
+878.42915,284,198.643241
+923.186142,321,199.135314
+848.057738,274,199.559453
+1269.439859,444,200.242507
+287.432224,136,200.451502
+972.058569,341,200.973242
+265.717732,136,201.176585
+847.980877,278,201.603978
+244.77217,129,201.802731
+231.113305,132,202.007009
+720.089566,266,202.415018
+779.332615,249,202.797632
+964.218065,299,203.259629
+666.265526,254,203.652119
+231.435681,126,203.842927
+962.207293,330,204.34968
+868.228811,299,204.809604
+897.951991,307,205.2826
+964.208795,311,205.761126
+263.795688,138,205.973092
+986.289281,295,206.430113
+245.409053,128,206.626628
+784.754304,263,207.039183
+1220.102511,419,207.682128
+1047.961391,340,208.210836
+766.928153,266,208.621729
+1034.044339,333,209.133857
+1197.946723,387,209.733588
+260.955835,128,209.930193
+1144.643221,368,210.49525
+861.323481,281,210.924232
+266.304395,134,211.131347
+250.16957,132,211.334835
+699.198502,253,211.726371
+243.311395,131,211.930995
+276.080638,136,212.140617
+929.624131,303,212.610336
+1446.888679,465,213.320601
+808.626526,276,213.749104
+240.304918,124,213.939471
+758.226254,236,214.299534
+1007.057343,315,214.775364
+901.32256,289,215.216009
+779.778507,264,215.624806
+1090.404018,357,216.173853
+269.382099,134,216.379939
+996.489995,311,216.856791
+1007.736893,324,217.356003
+918.393869,307,217.815643
+238.427475,131,218.015015
+737.878519,262,218.41623
+997.047305,335,218.921157
+885.30207,296,219.375606
+1333.020934,425,220.023831
+228.786988,126,220.214793
+890.261444,284,220.652377
+926.651139,302,221.111233
+1030.863,325,221.609219
+970.941288,318,222.093538
+855.593443,286,222.53772
+1145.754536,421,223.182185
+686.368054,255,223.572323
+272.727025,135,223.777847
+1026.331743,347,224.311628
+880.644257,285,224.747173
+920.550963,304,225.214821
+291.256843,138,225.427743
+936.562142,297,225.88005
+223.09637,128,226.07401
+819.576091,281,226.504897
+245.756956,133,226.706502
+761.538017,234,227.066893
+261.268062,133,227.269621
+285.532083,136,227.477694
+987.762681,326,227.971255
+1140.220795,371,228.539086
diff --git a/logs/benchmark/a2c-Swimmer-v3/0.monitor.csv b/logs/benchmark/a2c-Swimmer-v3/0.monitor.csv
new file mode 100644
index 000000000..4082b9b0f
--- /dev/null
+++ b/logs/benchmark/a2c-Swimmer-v3/0.monitor.csv
@@ -0,0 +1,152 @@
+#{"t_start": 1637100618.9998503, "env_id": "Swimmer-v3"}
+r,l,t
+200.140515,1000,4.242085
+198.203685,1000,5.736353
+199.360422,1000,7.238446
+199.808547,1000,8.731599
+201.445025,1000,10.234066
+199.876653,1000,11.72367
+198.097553,1000,13.22586
+201.586617,1000,14.711929
+198.614489,1000,16.196195
+201.928772,1000,17.692023
+199.494948,1000,19.184886
+202.864756,1000,20.682906
+200.947335,1000,22.168917
+203.294253,1000,23.666169
+201.67397,1000,25.160051
+199.091391,1000,26.645751
+200.702852,1000,28.135248
+200.510623,1000,29.626992
+200.72251,1000,31.12286
+198.478756,1000,32.607112
+205.192921,1000,34.102584
+199.045657,1000,35.597304
+204.14963,1000,37.093337
+199.067561,1000,38.590185
+200.442803,1000,40.020542
+203.590749,1000,41.513059
+201.695927,1000,43.008753
+201.876769,1000,44.499117
+200.592252,1000,46.000355
+198.229794,1000,47.48953
+199.592975,1000,48.970034
+213.024245,1000,50.462343
+199.698682,1000,51.94894
+200.414401,1000,53.447006
+197.823415,1000,54.936721
+198.264044,1000,56.440186
+198.513124,1000,57.816354
+199.770674,1000,59.044554
+199.675184,1000,60.268464
+197.414461,1000,61.491188
+196.016077,1000,62.712397
+201.397107,1000,63.956144
+198.585154,1000,65.442658
+198.763493,1000,66.931257
+202.915557,1000,68.420636
+208.807034,1000,69.912281
+200.194117,1000,71.401966
+200.138138,1000,72.890975
+206.129094,1000,74.384219
+199.295381,1000,75.873395
+198.286242,1000,77.365539
+201.280032,1000,78.855077
+202.676505,1000,80.353294
+204.001602,1000,81.848674
+200.762069,1000,83.340474
+200.940484,1000,84.82967
+201.505074,1000,86.322861
+198.561457,1000,87.828744
+206.585112,1000,89.323436
+199.330837,1000,90.817043
+199.538588,1000,92.314189
+198.716347,1000,93.81354
+200.186626,1000,95.304065
+200.941464,1000,96.79389
+202.335522,1000,98.288934
+198.759977,1000,99.777784
+199.970358,1000,101.276507
+200.475811,1000,102.769677
+199.256222,1000,104.255011
+199.156468,1000,105.75235
+200.075331,1000,107.255245
+200.215628,1000,108.754497
+198.710878,1000,110.250869
+197.729846,1000,111.743039
+198.044211,1000,113.235557
+202.653753,1000,114.727603
+201.956201,1000,116.216987
+198.917998,1000,117.719968
+202.99987,1000,119.22234
+199.577984,1000,120.720538
+201.170684,1000,122.213988
+202.496429,1000,123.703055
+201.575108,1000,125.200595
+197.503503,1000,126.69544
+199.222409,1000,128.195614
+198.633504,1000,129.690701
+200.094587,1000,131.190785
+204.45861,1000,132.680735
+205.129652,1000,134.173821
+198.70643,1000,135.522054
+198.957188,1000,137.027807
+200.037884,1000,138.520951
+200.218999,1000,140.021116
+198.438764,1000,141.516362
+199.853588,1000,143.008185
+198.502304,1000,144.505784
+203.056392,1000,146.000366
+201.203906,1000,147.490095
+198.113588,1000,148.988931
+201.082178,1000,150.491045
+198.403694,1000,151.992154
+199.800939,1000,153.492148
+199.761858,1000,154.994446
+197.296077,1000,156.498173
+199.330693,1000,157.985116
+199.120013,1000,159.47514
+198.670725,1000,160.983279
+202.220764,1000,162.485674
+197.871546,1000,163.832875
+202.051546,1000,165.055995
+201.015528,1000,166.274827
+199.869299,1000,167.494134
+197.697979,1000,168.715175
+198.521683,1000,169.942096
+201.341834,1000,171.165057
+198.464566,1000,172.381475
+208.994951,1000,173.605548
+198.531592,1000,175.092656
+204.824943,1000,176.601208
+199.635364,1000,178.100652
+200.694621,1000,179.594237
+200.879825,1000,181.084819
+198.689667,1000,182.57356
+200.554994,1000,184.074973
+200.866416,1000,185.57253
+202.089683,1000,187.061163
+198.98747,1000,188.552679
+198.571695,1000,190.044914
+205.703098,1000,191.537216
+200.21437,1000,193.035058
+200.476087,1000,194.520963
+202.790959,1000,196.010253
+199.464069,1000,197.510389
+200.279985,1000,198.998553
+199.55283,1000,200.491466
+199.593341,1000,201.989568
+200.059799,1000,203.484314
+201.175811,1000,204.97689
+199.905182,1000,206.473033
+200.741707,1000,207.969262
+199.303372,1000,209.470648
+198.332849,1000,210.974478
+204.846305,1000,212.472091
+207.226916,1000,213.974015
+198.778,1000,215.474386
+201.286953,1000,216.980722
+198.061316,1000,218.485187
+206.690535,1000,219.988058
+198.590361,1000,221.481822
+206.460309,1000,222.978997
diff --git a/logs/benchmark/benchmark.md b/logs/benchmark/benchmark.md
index 8d38d915d..e37e8b000 100644
--- a/logs/benchmark/benchmark.md
+++ b/logs/benchmark/benchmark.md
@@ -26,7 +26,9 @@ and also allow users to have access to pretrained agents.*
 |a2c  |BreakoutNoFrameskip-v4     |    279.793|   122.177|10M        |        604115|           82|
 |a2c  |CartPole-v1                |    500.000|     0.000|500k       |        150000|          300|
 |a2c  |EnduroNoFrameskip-v4       |      0.000|     0.000|10M        |        599040|           45|
+|a2c  |HalfCheetah-v3             |   3041.174|   157.265|1M         |        150000|          150|
 |a2c  |HalfCheetahBulletEnv-v0    |   2107.384|    36.008|2M         |        150000|          150|
+|a2c  |Hopper-v3                  |    733.454|   376.574|1M         |        149987|          580|
 |a2c  |HopperBulletEnv-v0         |    815.355|   313.798|2M         |        149541|          254|
 |a2c  |LunarLander-v2             |    155.751|    80.419|200k       |        149443|          297|
 |a2c  |LunarLanderContinuous-v2   |     84.225|   145.906|5M         |        149305|          256|
@@ -39,6 +41,7 @@ and also allow users to have access to pretrained agents.*
 |a2c  |RoadRunnerNoFrameskip-v4   |  31671.512|  6364.085|10M        |        606710|          172|
 |a2c  |SeaquestNoFrameskip-v4     |   1721.493|   105.339|10M        |        599691|           67|
 |a2c  |SpaceInvadersNoFrameskip-v4|    627.160|   201.974|10M        |        604848|          162|
+|a2c  |Swimmer-v3                 |    200.627|     2.544|1M         |        150000|          150|
 |a2c  |Walker2DBulletEnv-v0       |    858.209|   333.116|2M         |        149156|          173|
 |ddpg |AntBulletEnv-v0            |   2399.147|    75.410|1M         |        150000|          150|
 |ddpg |BipedalWalker-v3           |    197.486|   141.580|1M         |        149237|          227|
@@ -63,6 +66,7 @@ and also allow users to have access to pretrained agents.*
 |dqn  |SeaquestNoFrameskip-v4     |   2000.290|   606.644|10M        |        599505|           69|
 |dqn  |SpaceInvadersNoFrameskip-v4|    622.742|   201.564|10M        |        604311|          155|
 |ppo  |Acrobot-v1                 |    -73.506|    18.201|1M         |        149979|         2013|
+|ppo  |Ant-v3                     |   1327.158|   451.577|1M         |        149572|          175|
 |ppo  |AntBulletEnv-v0            |   2865.922|    56.468|2M         |        150000|          150|
 |ppo  |AsteroidsNoFrameskip-v4    |   2156.174|   744.640|10M        |        602092|          149|
 |ppo  |BeamRiderNoFrameskip-v4    |   3397.000|  1662.368|10M        |        598926|           46|
@@ -71,7 +75,9 @@ and also allow users to have access to pretrained agents.*
 |ppo  |BreakoutNoFrameskip-v4     |    398.033|    33.328|10M        |        600418|           60|
 |ppo  |CartPole-v1                |    500.000|     0.000|100k       |        150000|          300|
 |ppo  |EnduroNoFrameskip-v4       |    996.364|   176.090|10M        |        572416|           11|
+|ppo  |HalfCheetah-v3             |   5819.099|   663.530|1M         |        150000|          150|
 |ppo  |HalfCheetahBulletEnv-v0    |   2924.721|    64.465|2M         |        150000|          150|
+|ppo  |Hopper-v3                  |   2410.435|    10.026|1M         |        150000|          150|
 |ppo  |HopperBulletEnv-v0         |   2575.054|   223.301|2M         |        149094|          152|
 |ppo  |LunarLander-v2             |    242.119|    31.823|1M         |        149636|          369|
 |ppo  |LunarLanderContinuous-v2   |    270.863|    32.072|1M         |        149956|          526|
@@ -84,7 +90,9 @@ and also allow users to have access to pretrained agents.*
 |ppo  |RoadRunnerNoFrameskip-v4   |  40680.645|  6675.058|10M        |        605786|          155|
 |ppo  |SeaquestNoFrameskip-v4     |   1783.636|    34.096|10M        |        598243|           66|
 |ppo  |SpaceInvadersNoFrameskip-v4|    960.331|   425.355|10M        |        603771|          136|
+|ppo  |Swimmer-v3                 |    281.561|     9.671|1M         |        150000|          150|
 |ppo  |Walker2DBulletEnv-v0       |   2109.992|    13.899|2M         |        150000|          150|
+|ppo  |Walker2d-v3                |   3478.798|   821.708|1M         |        149343|          171|
 |qrdqn|Acrobot-v1                 |    -69.135|     9.967|100k       |        149949|         2138|
 |qrdqn|AsteroidsNoFrameskip-v4    |   2185.303|  1097.172|10M        |        599784|           66|
 |qrdqn|BeamRiderNoFrameskip-v4    |  17122.941| 10769.997|10M        |        596483|           17|
@@ -98,26 +106,39 @@ and also allow users to have access to pretrained agents.*
 |qrdqn|RoadRunnerNoFrameskip-v4   |  42325.424|  8361.161|10M        |        591016|           59|
 |qrdqn|SeaquestNoFrameskip-v4     |   2557.576|    76.951|10M        |        596275|           66|
 |qrdqn|SpaceInvadersNoFrameskip-v4|   1899.928|   823.488|10M        |        597218|           69|
+|sac  |Ant-v3                     |   4615.791|  1354.111|1M         |        149074|          165|
 |sac  |AntBulletEnv-v0            |   3073.114|   175.148|1M         |        150000|          150|
 |sac  |BipedalWalker-v3           |    297.668|    33.060|500k       |        149530|          136|
 |sac  |BipedalWalkerHardcore-v3   |      4.423|   103.910|10M        |        149794|           88|
+|sac  |HalfCheetah-v3             |   9535.451|   100.470|1M         |        150000|          150|
 |sac  |HalfCheetahBulletEnv-v0    |   2792.170|    12.088|1M         |        150000|          150|
+|sac  |Hopper-v3                  |   2325.547|  1129.676|1M         |        149841|          236|
 |sac  |HopperBulletEnv-v0         |   2603.494|   164.322|1M         |        149724|          151|
+|sac  |Humanoid-v3                |   6232.287|   279.885|2M         |        149460|          150|
 |sac  |LunarLanderContinuous-v2   |    260.390|    65.467|500k       |        149634|          672|
 |sac  |MountainCarContinuous-v0   |     94.679|     1.134|50k        |        149966|         1443|
 |sac  |Pendulum-v0                |   -156.995|    88.714|20k        |        150000|          750|
 |sac  |ReacherBulletEnv-v0        |     18.062|     9.729|300k       |        150000|         1000|
+|sac  |Swimmer-v3                 |    345.568|     3.084|1M         |        150000|          150|
 |sac  |Walker2DBulletEnv-v0       |   2292.266|    13.970|1M         |        149983|          150|
+|sac  |Walker2d-v3                |   3863.203|   254.347|1M         |        149309|          150|
+|td3  |Ant-v3                     |   5813.274|   589.773|1M         |        149393|          151|
 |td3  |AntBulletEnv-v0            |   3300.026|    54.640|1M         |        150000|          150|
 |td3  |BipedalWalker-v3           |    305.990|    56.886|1M         |        149999|          224|
 |td3  |BipedalWalkerHardcore-v3   |    -98.116|    16.087|10M        |        150000|           75|
+|td3  |HalfCheetah-v3             |   9655.666|   969.916|1M         |        150000|          150|
 |td3  |HalfCheetahBulletEnv-v0    |   2821.641|    19.722|1M         |        150000|          150|
+|td3  |Hopper-v3                  |   3606.390|     4.027|1M         |        150000|          150|
 |td3  |HopperBulletEnv-v0         |   2681.609|    27.806|1M         |        149486|          150|
+|td3  |Humanoid-v3                |   5566.687|    14.544|2M         |        150000|          150|
 |td3  |LunarLanderContinuous-v2   |    207.451|    67.562|300k       |        149488|          337|
 |td3  |MountainCarContinuous-v0   |     93.483|     0.075|300k       |        149976|         2275|
 |td3  |Pendulum-v0                |   -151.855|    90.227|20k        |        150000|          750|
 |td3  |ReacherBulletEnv-v0        |     17.114|     9.750|300k       |        150000|         1000|
+|td3  |Swimmer-v3                 |    359.127|     1.244|1M         |        150000|          150|
 |td3  |Walker2DBulletEnv-v0       |   2213.672|   230.558|1M         |        149800|          152|
+|td3  |Walker2d-v3                |   4717.823|    46.303|1M         |        150000|          150|
+|tqc  |Ant-v3                     |   3339.362|  1969.906|1M         |        149583|          202|
 |tqc  |AntBulletEnv-v0            |   3456.717|   248.733|1M         |        150000|          150|
 |tqc  |BipedalWalker-v3           |    329.808|    45.083|500k       |        149682|          254|
 |tqc  |BipedalWalkerHardcore-v3   |    235.226|   110.569|2M         |        149032|          131|
@@ -125,8 +146,11 @@ and also allow users to have access to pretrained agents.*
 |tqc  |FetchPush-v1               |     -8.799|     5.438|1M         |        150000|         3000|
 |tqc  |FetchReach-v1              |     -1.659|     0.873|20k        |        150000|         3000|
 |tqc  |FetchSlide-v1              |    -29.210|    11.387|3M         |        150000|         3000|
+|tqc  |HalfCheetah-v3             |  12089.939|   127.440|1M         |        150000|          150|
 |tqc  |HalfCheetahBulletEnv-v0    |   3675.299|    17.681|1M         |        150000|          150|
+|tqc  |Hopper-v3                  |   3754.199|     8.276|1M         |        150000|          150|
 |tqc  |HopperBulletEnv-v0         |   2662.373|   206.210|1M         |        149881|          151|
+|tqc  |Humanoid-v3                |   7239.320|  1647.498|2M         |        149508|          165|
 |tqc  |LunarLanderContinuous-v2   |    277.956|    25.466|500k       |        149928|          706|
 |tqc  |MountainCarContinuous-v0   |     63.641|    45.259|50k        |        149796|          186|
 |tqc  |PandaPickAndPlace-v1       |     -8.024|     6.674|1M         |        150000|         3000|
@@ -136,5 +160,7 @@ and also allow users to have access to pretrained agents.*
 |tqc  |PandaStack-v1              |    -96.915|    17.240|1M         |        150000|         1500|
 |tqc  |Pendulum-v0                |   -151.340|    87.893|20k        |        150000|          750|
 |tqc  |ReacherBulletEnv-v0        |     18.255|     9.543|300k       |        150000|         1000|
+|tqc  |Swimmer-v3                 |    339.423|     1.486|1M         |        150000|          150|
 |tqc  |Walker2DBulletEnv-v0       |   2508.934|   614.624|1M         |        149572|          159|
+|tqc  |Walker2d-v3                |   4380.720|   500.489|1M         |        149606|          152|
 |tqc  |parking-v0                 |     -6.762|     2.690|100k       |        149983|         7528|
diff --git a/logs/benchmark/ppo-Ant-v3/0.monitor.csv b/logs/benchmark/ppo-Ant-v3/0.monitor.csv
new file mode 100644
index 000000000..2dead6b00
--- /dev/null
+++ b/logs/benchmark/ppo-Ant-v3/0.monitor.csv
@@ -0,0 +1,177 @@
+#{"t_start": 1635520729.8010325, "env_id": "Ant-v3"}
+r,l,t
+1840.882195,1000,1.113703
+1363.011461,1000,2.195891
+1491.364542,1000,3.258408
+882.810055,548,3.842525
+1239.138363,662,4.54904
+1713.685042,1000,5.59651
+1846.824304,1000,6.625901
+1623.226618,955,7.586072
+1739.239769,1000,8.598306
+685.925705,1000,9.697511
+1904.408651,1000,10.775355
+1728.251425,1000,11.831213
+1008.582267,537,12.359337
+1443.955235,792,13.146892
+1765.968762,1000,14.186844
+21.27447,19,14.205679
+1729.17021,1000,15.238421
+1285.805522,724,15.95481
+1710.628279,1000,17.014316
+695.145173,391,17.410278
+1747.661093,1000,18.432171
+1371.931285,1000,19.53008
+1742.417777,1000,20.580762
+1822.31459,1000,21.598843
+1770.172315,1000,22.660461
+1665.838951,1000,23.696963
+1762.113634,1000,24.800067
+1586.876946,1000,25.892547
+521.874437,296,26.211875
+1057.95211,551,26.779757
+1580.588225,1000,27.808496
+900.178797,548,28.388348
+1237.125501,1000,29.454114
+1768.891527,1000,30.489027
+1394.209672,1000,31.520578
+1746.430391,1000,33.039368
+607.986989,344,33.498292
+1710.482771,1000,34.582572
+1888.720093,1000,35.708852
+1175.617736,632,36.356046
+48.06013,42,36.399422
+1655.967344,1000,37.447802
+1667.293515,1000,38.480164
+1681.848949,1000,39.467385
+1715.066634,1000,40.726793
+770.428194,413,41.237572
+1184.42587,632,42.325856
+855.019966,1000,43.786138
+1849.875054,1000,44.960639
+1141.499227,629,45.614417
+1692.779239,1000,46.738504
+1274.042874,1000,47.808586
+1704.931667,1000,49.324414
+1597.573691,1000,51.096245
+816.552059,483,51.675364
+978.032491,1000,53.157686
+654.988698,400,53.830339
+1687.50206,1000,55.797084
+1737.825578,1000,58.467178
+911.568601,505,59.746958
+1658.336763,1000,61.524154
+1660.766534,1000,62.65543
+227.230183,172,62.830936
+1797.112781,1000,63.963328
+1779.563752,1000,65.233653
+1038.301071,535,65.933947
+1708.953416,1000,67.123411
+385.282324,231,67.359632
+1726.60275,1000,68.356907
+1771.137912,1000,69.327081
+1046.933672,1000,70.481815
+1336.742002,1000,71.678807
+928.530401,1000,72.860141
+1703.630429,1000,73.933302
+1224.950503,1000,75.029214
+1242.349443,1000,76.114613
+736.970356,1000,77.23227
+1156.294941,1000,78.36749
+1279.061078,1000,79.414185
+1015.509492,1000,80.55673
+781.365243,495,81.245426
+1185.401862,680,82.433879
+920.735581,1000,83.76754
+909.526343,1000,85.067804
+1469.483378,1000,86.085187
+1866.954391,1000,87.297341
+959.090962,1000,88.842965
+1319.014894,1000,90.113227
+188.848298,122,90.237763
+1078.886652,1000,91.239151
+1738.048141,1000,92.228588
+1606.720917,1000,93.237816
+1101.49143,597,93.904159
+1599.09518,1000,94.975118
+1722.908261,1000,95.995843
+1592.404561,1000,97.126825
+1490.642607,1000,98.490684
+1771.758535,1000,100.039138
+1773.745448,1000,101.067975
+1647.419625,1000,102.166967
+877.53996,1000,103.469107
+661.041801,370,103.913144
+834.265656,1000,105.135507
+1708.00049,1000,106.406227
+1873.513883,1000,107.552981
+1781.471992,1000,108.566889
+1136.359062,640,109.228695
+777.655251,498,109.765946
+929.102799,519,110.277192
+1764.142398,1000,111.283007
+1844.383777,1000,112.341875
+1659.502055,1000,113.381933
+1616.016703,1000,114.39198
+1332.177249,1000,115.438612
+1734.882415,1000,116.438935
+1893.387213,1000,117.401114
+704.922304,387,117.771293
+1797.320005,1000,118.755551
+1723.101305,1000,119.737746
+863.867063,1000,120.771125
+1518.072112,1000,121.743454
+1886.234069,1000,122.774438
+1631.245577,1000,123.779782
+1755.923335,1000,124.816073
+940.238529,535,125.338921
+1712.5147,1000,126.30922
+1822.615509,1000,127.321775
+938.15586,510,127.840631
+1714.105949,1000,128.874546
+1006.088875,1000,129.90647
+932.517264,1000,131.000208
+1209.586947,684,131.704403
+860.500415,1000,132.893804
+986.679501,1000,133.932454
+920.783691,1000,135.00164
+707.331407,369,135.421159
+1292.008063,1000,136.475228
+1011.054566,1000,137.591502
+1644.900928,1000,138.65183
+905.833023,500,139.170008
+1258.483305,1000,140.219795
+1410.765431,808,141.05081
+276.316837,184,141.233557
+1351.48226,1000,142.285585
+1007.18965,1000,143.315243
+1321.842512,717,144.072212
+768.084683,480,144.574024
+1707.578863,1000,145.665311
+952.576179,1000,146.779233
+932.276904,1000,147.842505
+1687.426028,1000,148.89482
+1722.561402,923,149.815121
+595.753801,311,150.124333
+1776.836862,1000,151.147339
+1644.62719,1000,152.169931
+1295.849745,1000,153.173975
+1227.360715,1000,154.198111
+1769.958273,1000,155.227583
+663.794596,387,155.623412
+1633.56957,1000,156.633234
+1573.881873,1000,157.701624
+885.937994,540,158.267609
+1558.796041,1000,159.285657
+1785.594843,1000,160.330084
+1050.781416,584,160.929931
+1827.872685,1000,161.916903
+1689.767878,1000,162.913024
+1747.810022,1000,163.892219
+1827.338328,1000,164.918605
+1713.81547,1000,165.991497
+317.803709,210,166.223451
+591.361068,362,166.59217
+1205.919233,1000,167.666028
+463.739535,278,167.940594
+1547.137558,841,168.766441
diff --git a/logs/benchmark/ppo-HalfCheetah-v3/0.monitor.csv b/logs/benchmark/ppo-HalfCheetah-v3/0.monitor.csv
new file mode 100644
index 000000000..3f6426f83
--- /dev/null
+++ b/logs/benchmark/ppo-HalfCheetah-v3/0.monitor.csv
@@ -0,0 +1,152 @@
+#{"t_start": 1637099942.3303025, "env_id": "HalfCheetah-v3"}
+r,l,t
+5938.860024,1000,3.666764
+5935.924158,1000,4.728658
+5667.517046,1000,5.789934
+5916.997203,1000,6.854891
+5897.189085,1000,7.916115
+5924.30196,1000,8.979527
+5921.135028,1000,10.041186
+5907.297499,1000,11.101725
+5373.971602,1000,12.161371
+5879.464644,1000,13.222286
+5858.928513,1000,14.281779
+5929.889604,1000,15.342736
+5951.033711,1000,16.402832
+5927.927649,1000,17.464748
+5913.955385,1000,18.526745
+5957.28893,1000,19.584906
+5888.91279,1000,20.648084
+5885.412088,1000,21.710864
+5848.715745,1000,22.771638
+5920.962025,1000,23.830917
+5881.56609,1000,24.891874
+5902.744111,1000,25.951776
+5895.550477,1000,27.01202
+5926.274267,1000,28.071614
+5910.030981,1000,29.130269
+5914.811302,1000,30.187297
+5916.440007,1000,31.252222
+5917.573268,1000,32.293522
+5923.294646,1000,33.386748
+5909.600501,1000,34.252114
+5927.693111,1000,35.141572
+5940.486317,1000,36.402054
+5934.155869,1000,37.921134
+5922.484878,1000,39.436687
+5912.867114,1000,40.951943
+5901.844951,1000,42.461932
+5934.68604,1000,43.97436
+5958.49846,1000,45.487876
+5918.759423,1000,46.993566
+5910.519325,1000,48.503974
+5909.574441,1000,50.016741
+5900.632548,1000,51.536162
+5923.33169,1000,53.047302
+5903.78039,1000,54.55674
+5947.755564,1000,56.068441
+5767.523761,1000,57.578068
+5929.020415,1000,59.098892
+5931.389266,1000,60.607038
+5894.740934,1000,62.11791
+5896.769641,1000,63.628218
+5870.783803,1000,65.140182
+5899.732507,1000,66.65238
+5888.83461,1000,68.172634
+5681.628637,1000,69.686076
+5926.568678,1000,71.202078
+5908.317013,1000,72.712083
+5960.003192,1000,74.223531
+5907.804824,1000,75.729294
+5888.02769,1000,77.240194
+5932.007117,1000,78.750913
+5907.62341,1000,80.272232
+5879.394881,1000,81.780397
+5910.721587,1000,83.294829
+5935.562086,1000,84.794854
+5932.203592,1000,86.302147
+5922.534624,1000,87.815541
+5918.091731,1000,89.32249
+5934.159058,1000,90.833594
+5895.503822,1000,92.340074
+5950.468455,1000,93.854977
+5869.272552,1000,95.372555
+5933.268171,1000,96.892232
+5957.47103,1000,98.408154
+5929.946201,1000,99.920677
+5958.077883,1000,101.436747
+5929.530279,1000,102.956238
+5954.880563,1000,104.465036
+5922.823871,1000,105.976162
+5922.038835,1000,107.490245
+5927.870576,1000,109.006176
+5976.576639,1000,110.509937
+5934.166076,1000,112.011514
+5933.31148,1000,113.496559
+5936.019072,1000,114.988856
+5924.932396,1000,116.473086
+296.358303,1000,117.962865
+5938.348806,1000,119.452364
+5891.474921,1000,120.951307
+5914.538071,1000,122.458736
+5912.975342,1000,123.956698
+5923.951017,1000,125.44763
+5945.21555,1000,126.944135
+5950.535387,1000,128.434389
+5932.832453,1000,129.926241
+5932.976912,1000,131.407532
+5941.83891,1000,132.669448
+5912.305846,1000,133.938746
+5906.888593,1000,135.387641
+5930.029177,1000,136.797336
+5903.766295,1000,138.281872
+5938.83975,1000,139.76903
+5934.743554,1000,141.262157
+5958.062925,1000,142.751768
+5867.453371,1000,144.249911
+5955.11245,1000,145.742555
+5896.92179,1000,147.237629
+5883.251442,1000,148.731834
+5951.866345,1000,150.222506
+5910.22188,1000,151.71331
+5935.151047,1000,153.211881
+5909.428332,1000,154.701092
+5905.277642,1000,156.200911
+5921.405911,1000,157.689279
+5923.759658,1000,159.186161
+1731.170084,1000,160.678878
+5929.640294,1000,162.162521
+5954.04537,1000,163.661633
+5942.365224,1000,165.153059
+5913.358098,1000,166.648004
+5905.566914,1000,168.136005
+5939.551402,1000,169.633024
+5934.39503,1000,171.123961
+5948.703334,1000,172.606333
+1693.113236,1000,174.108766
+5845.579733,1000,175.597616
+5916.69902,1000,177.092502
+5941.612002,1000,178.581881
+5868.869912,1000,180.07258
+5917.851532,1000,181.566799
+5923.916291,1000,183.058157
+5878.231378,1000,184.547096
+5958.955404,1000,186.03087
+5892.624759,1000,187.521097
+5933.15034,1000,189.012439
+5907.681947,1000,190.503134
+5929.410253,1000,191.992864
+5924.534991,1000,193.487819
+5949.782112,1000,194.973958
+5937.646126,1000,196.468887
+5937.882353,1000,197.962804
+5952.860534,1000,199.45123
+5901.797173,1000,200.953263
+5919.561938,1000,202.447112
+5901.521361,1000,203.934131
+5891.564351,1000,205.422009
+5922.248593,1000,206.917509
+5978.464793,1000,208.40654
+5960.432074,1000,209.897258
+5924.42939,1000,211.385628
+5959.832146,1000,212.872656
diff --git a/logs/benchmark/ppo-Hopper-v3/0.monitor.csv b/logs/benchmark/ppo-Hopper-v3/0.monitor.csv
new file mode 100644
index 000000000..36399faca
--- /dev/null
+++ b/logs/benchmark/ppo-Hopper-v3/0.monitor.csv
@@ -0,0 +1,152 @@
+#{"t_start": 1637100157.4684572, "env_id": "Hopper-v3"}
+r,l,t
+2397.917399,1000,4.312675
+2415.857381,1000,5.858377
+2408.418643,1000,7.404146
+2400.762199,1000,8.946695
+2418.952303,1000,10.500146
+2427.637905,1000,12.053664
+2400.862324,1000,13.474802
+2404.969555,1000,14.993221
+2403.148149,1000,16.540131
+2422.531931,1000,18.082354
+2399.646816,1000,19.631776
+2418.94744,1000,21.145653
+2485.345696,1000,22.687819
+2411.912902,1000,24.232273
+2420.83685,1000,25.776517
+2395.014679,1000,27.316612
+2417.487406,1000,28.870345
+2415.889398,1000,30.414967
+2420.701709,1000,31.964652
+2409.261642,1000,33.514201
+2420.646473,1000,35.061497
+2404.28479,1000,36.612007
+2408.569326,1000,38.164888
+2398.986396,1000,39.716375
+2414.51933,1000,41.257097
+2404.19985,1000,42.808993
+2408.057923,1000,44.35624
+2419.527801,1000,45.898386
+2402.386435,1000,47.436276
+2397.89634,1000,48.981411
+2395.325083,1000,50.525915
+2402.206866,1000,52.072796
+2415.508497,1000,53.615285
+2402.307157,1000,55.163954
+2409.709897,1000,56.710325
+2417.652624,1000,58.260012
+2404.195035,1000,59.809412
+2408.643871,1000,61.365802
+2404.869774,1000,62.908635
+2413.380626,1000,64.450398
+2418.858282,1000,65.992393
+2410.264573,1000,67.531456
+2402.895379,1000,69.071171
+2404.073425,1000,70.612782
+2409.294848,1000,72.160056
+2403.178955,1000,73.711461
+2411.823278,1000,75.250633
+2414.994635,1000,76.795025
+2424.850557,1000,78.33114
+2406.281315,1000,79.875129
+2420.157425,1000,81.426608
+2413.999947,1000,82.969026
+2415.762779,1000,84.506117
+2418.614528,1000,86.051751
+2420.915654,1000,87.602057
+2416.255039,1000,89.144122
+2412.413825,1000,90.685082
+2414.04466,1000,92.221291
+2394.896588,1000,93.769446
+2407.689659,1000,95.310808
+2406.635945,1000,96.859557
+2401.613874,1000,98.401602
+2425.986087,1000,99.945781
+2418.490523,1000,101.486911
+2416.237503,1000,103.030096
+2401.808563,1000,104.574748
+2407.424644,1000,106.114195
+2418.447939,1000,107.655586
+2399.128648,1000,109.091639
+2406.259413,1000,110.392413
+2397.757461,1000,111.83158
+2420.812275,1000,113.370616
+2402.751583,1000,114.912548
+2415.923214,1000,116.441404
+2406.571922,1000,117.982879
+2402.971876,1000,119.495638
+2403.508647,1000,121.039729
+2426.647026,1000,122.576833
+2403.7589,1000,124.12054
+2403.321882,1000,125.665254
+2415.927475,1000,127.216013
+2413.000124,1000,128.75558
+2405.814898,1000,130.300268
+2413.843345,1000,131.83469
+2413.729198,1000,133.377843
+2427.920201,1000,134.918779
+2413.497153,1000,136.472244
+2402.863698,1000,138.013369
+2409.90538,1000,139.560426
+2422.178063,1000,141.108708
+2404.975574,1000,142.64835
+2402.164061,1000,144.207044
+2420.850014,1000,145.756333
+2424.484151,1000,147.30708
+2424.496995,1000,148.856881
+2414.634545,1000,150.398678
+2398.451116,1000,151.946911
+2417.94316,1000,153.497376
+2403.316716,1000,155.051297
+2413.177329,1000,156.597595
+2419.458847,1000,158.14412
+2420.179153,1000,159.688891
+2416.5654,1000,161.24047
+2399.535405,1000,162.799268
+2414.085116,1000,164.346028
+2404.489404,1000,165.894187
+2416.988641,1000,167.441612
+2407.067824,1000,168.982146
+2414.833,1000,170.525674
+2417.794511,1000,172.07214
+2410.336162,1000,173.620674
+2416.536334,1000,175.154684
+2412.840003,1000,176.702632
+2416.495633,1000,178.242066
+2404.710519,1000,179.782672
+2413.046347,1000,181.335237
+2414.221228,1000,182.877237
+2413.947165,1000,184.413441
+2407.9539,1000,185.962509
+2404.188584,1000,187.508194
+2405.221548,1000,189.04655
+2401.644287,1000,190.587839
+2405.66172,1000,192.129124
+2408.805605,1000,193.680422
+2395.091946,1000,195.230978
+2401.645416,1000,196.775101
+2405.650844,1000,198.319307
+2419.54914,1000,199.858149
+2408.435649,1000,201.400156
+2400.996891,1000,202.940157
+2399.933281,1000,204.479447
+2418.418065,1000,206.012829
+2410.187469,1000,207.318651
+2406.35426,1000,208.637513
+2405.722459,1000,210.181823
+2406.169715,1000,211.727912
+2414.52326,1000,213.275121
+2421.004243,1000,214.816169
+2405.615296,1000,216.358583
+2404.352381,1000,217.862846
+2399.704569,1000,219.394016
+2403.652643,1000,220.951306
+2403.394861,1000,222.493833
+2395.616625,1000,224.047565
+2403.123076,1000,225.587986
+2416.802082,1000,227.135288
+2403.126001,1000,228.681602
+2402.579348,1000,230.219259
+2402.209814,1000,231.759451
+2405.275865,1000,233.302031
diff --git a/logs/benchmark/ppo-Swimmer-v3/0.monitor.csv b/logs/benchmark/ppo-Swimmer-v3/0.monitor.csv
new file mode 100644
index 000000000..f13bdfa90
--- /dev/null
+++ b/logs/benchmark/ppo-Swimmer-v3/0.monitor.csv
@@ -0,0 +1,152 @@
+#{"t_start": 1635520902.2598746, "env_id": "Swimmer-v3"}
+r,l,t
+282.266104,1000,0.646776
+290.914403,1000,1.254681
+284.434505,1000,1.867859
+283.242547,1000,2.475826
+286.74581,1000,3.098025
+274.629467,1000,3.709191
+286.012419,1000,4.29662
+283.876434,1000,4.881782
+287.939286,1000,5.486109
+288.94996,1000,6.07702
+289.533777,1000,6.676216
+290.791418,1000,7.270571
+285.030437,1000,7.877868
+286.657144,1000,8.517455
+281.51947,1000,9.154811
+287.189558,1000,9.759592
+283.475313,1000,10.34929
+289.104704,1000,10.946338
+287.07577,1000,11.613868
+256.064606,1000,12.248824
+271.668225,1000,12.875877
+290.325366,1000,13.514563
+287.149004,1000,14.111871
+285.549914,1000,14.71145
+288.279145,1000,15.313987
+283.685146,1000,15.954557
+276.314366,1000,16.575671
+287.949695,1000,17.16806
+280.703677,1000,17.780128
+287.099422,1000,18.381856
+284.404581,1000,18.969913
+287.798756,1000,19.534945
+283.697048,1000,20.131546
+284.319075,1000,20.734903
+288.957588,1000,21.347936
+255.366144,1000,21.954911
+289.937542,1000,22.566858
+282.164541,1000,23.154854
+285.932992,1000,23.777419
+269.298562,1000,24.395253
+289.309912,1000,24.988462
+279.654466,1000,25.578026
+289.080731,1000,26.179317
+282.81351,1000,26.793354
+266.149351,1000,27.377842
+282.561672,1000,27.954689
+278.350714,1000,28.537235
+281.611352,1000,29.136526
+267.458007,1000,29.748905
+289.490455,1000,30.335608
+285.450397,1000,30.911634
+287.302402,1000,31.513137
+284.438287,1000,32.09877
+288.74994,1000,32.740423
+289.00955,1000,33.386459
+288.639827,1000,34.018199
+282.908255,1000,34.607603
+286.964666,1000,35.239841
+283.39746,1000,35.860252
+283.678916,1000,36.477633
+284.491598,1000,37.064569
+286.170233,1000,37.669639
+285.351417,1000,38.278789
+288.552798,1000,38.872721
+279.230543,1000,39.458491
+287.563933,1000,40.055366
+275.704663,1000,40.697424
+258.907539,1000,41.340423
+287.399959,1000,41.940084
+286.357094,1000,42.534546
+256.336351,1000,43.119244
+265.847553,1000,43.72195
+288.765021,1000,44.30962
+251.659054,1000,44.888242
+280.076943,1000,45.456377
+286.469593,1000,46.023413
+254.171626,1000,46.609399
+286.324946,1000,47.190511
+288.206314,1000,47.780868
+281.115762,1000,48.373294
+288.048085,1000,48.99022
+288.753061,1000,49.641439
+288.937795,1000,50.27239
+281.813532,1000,50.861821
+285.718258,1000,51.438933
+283.387527,1000,52.020952
+283.153058,1000,52.591664
+281.598978,1000,53.157808
+287.143291,1000,53.724298
+284.172437,1000,54.295277
+288.765506,1000,54.862607
+263.464563,1000,55.479105
+285.855905,1000,56.063462
+282.752616,1000,56.642117
+288.225058,1000,57.221401
+282.346609,1000,57.789558
+286.847639,1000,58.373932
+286.681143,1000,58.977726
+286.883981,1000,59.608976
+286.905017,1000,60.241269
+277.13863,1000,60.835245
+290.200074,1000,61.454793
+260.907942,1000,62.0766
+282.783022,1000,62.717286
+283.145098,1000,63.356989
+288.460476,1000,63.950026
+267.896854,1000,64.538238
+280.283729,1000,65.151874
+281.795944,1000,65.745228
+281.942632,1000,66.329131
+248.128612,1000,66.915662
+284.685584,1000,67.510193
+246.055863,1000,68.1309
+286.447979,1000,68.782621
+265.674512,1000,69.430577
+267.126281,1000,70.040874
+288.869763,1000,70.624788
+287.231493,1000,71.209183
+280.423925,1000,71.838914
+282.902407,1000,72.444231
+278.906943,1000,73.062863
+285.674219,1000,73.653624
+280.306914,1000,74.222391
+249.917882,1000,74.792029
+283.620049,1000,75.353692
+282.839192,1000,75.944507
+248.698787,1000,76.530462
+277.666715,1000,77.123716
+287.151355,1000,77.707366
+276.110552,1000,78.275946
+291.663336,1000,78.841065
+279.998604,1000,79.432765
+284.580828,1000,80.013133
+284.508189,1000,80.599197
+281.888561,1000,81.193876
+284.021676,1000,81.785081
+283.094554,1000,82.384191
+283.293688,1000,82.991371
+290.491091,1000,83.603079
+281.532552,1000,84.24835
+283.253383,1000,84.853181
+286.214703,1000,85.439769
+289.164511,1000,86.024898
+284.523948,1000,86.60377
+286.261083,1000,87.185215
+284.709644,1000,87.756693
+286.403807,1000,88.333531
+282.30313,1000,88.909028
+283.220615,1000,89.49498
+286.824658,1000,90.082787
diff --git a/logs/benchmark/ppo-Walker2d-v3/0.monitor.csv b/logs/benchmark/ppo-Walker2d-v3/0.monitor.csv
new file mode 100644
index 000000000..f866541ed
--- /dev/null
+++ b/logs/benchmark/ppo-Walker2d-v3/0.monitor.csv
@@ -0,0 +1,173 @@
+#{"t_start": 1635521084.3830566, "env_id": "Walker2d-v3"}
+r,l,t
+4026.210402,1000,0.697392
+3772.246222,1000,1.372572
+3060.875309,738,1.870155
+4088.203691,1000,2.535861
+3914.921856,1000,3.21353
+3852.011012,1000,3.881989
+3582.063578,867,4.461374
+3920.567991,1000,5.137424
+3967.009547,1000,5.806421
+3738.995107,929,6.455853
+3749.240417,1000,7.132656
+3393.557753,835,7.697701
+4042.680103,1000,8.36728
+3706.445789,919,8.984324
+3382.268937,842,9.589025
+803.203988,229,9.751861
+4152.457394,1000,10.449339
+2270.301715,556,10.836567
+3891.494763,1000,11.532977
+4149.033257,997,12.206564
+3822.586967,1000,12.883372
+3365.886559,823,13.434976
+4041.652634,1000,14.107446
+3882.526575,1000,14.78741
+3840.791527,1000,15.458411
+3941.997667,1000,16.134604
+3344.517191,841,16.712657
+3823.378948,1000,17.402117
+3828.109882,1000,18.117902
+4057.90951,1000,18.822287
+4103.866625,1000,19.503992
+3752.013597,1000,20.228919
+3918.017348,1000,20.92953
+4152.867133,1000,21.624854
+3972.923024,1000,22.321552
+4064.412193,1000,23.015092
+3846.959564,1000,23.704244
+4049.183788,1000,24.416284
+3925.706591,938,25.073459
+3179.610636,781,25.623175
+3353.230413,831,26.211322
+2356.766108,594,26.63291
+981.376102,268,26.820556
+3922.080651,943,27.485534
+3166.32484,782,28.034033
+3952.280451,959,28.691564
+3011.816215,744,29.189228
+3701.387088,910,29.812245
+4053.035433,997,30.484271
+3238.453279,792,31.036439
+4079.554,1000,31.738122
+3913.871636,1000,32.423682
+3844.83184,957,33.078149
+3390.445996,825,33.636166
+3409.417818,825,34.185944
+3946.685584,1000,34.855189
+3204.399628,773,35.377683
+4038.020364,1000,36.044792
+3058.874673,759,36.559767
+765.439824,215,36.704095
+3710.422258,902,37.305271
+3784.372788,937,37.940198
+2355.261325,611,38.348946
+3870.73679,1000,39.026603
+4036.348839,1000,39.720443
+2876.545427,712,40.222855
+2748.376335,690,40.711256
+3843.17339,1000,41.411784
+3932.773703,1000,42.114575
+3879.007248,1000,42.812047
+3935.028921,1000,43.495664
+3232.49867,803,44.034029
+1033.034474,287,44.231498
+2403.783289,605,44.641037
+2755.413302,697,45.114166
+2234.642965,576,45.508586
+3993.298964,1000,46.195718
+3654.149005,890,46.812092
+4020.296531,1000,47.51093
+2803.359839,696,47.988065
+3819.265772,1000,48.694713
+4089.561212,1000,49.390889
+2863.516231,715,49.877812
+3123.200816,779,50.419853
+3884.408952,986,51.10247
+798.66673,222,51.257603
+4027.365236,1000,51.937644
+2340.06391,593,52.345252
+4106.263792,1000,53.028834
+3873.243356,1000,53.729511
+4090.568819,1000,54.409554
+4030.057873,1000,55.076714
+2803.461525,697,55.666827
+3752.346018,1000,56.414561
+3918.906102,940,57.05787
+3873.032617,1000,57.747838
+3560.41787,885,58.475419
+1962.701124,523,58.912298
+3819.608666,1000,60.227693
+3045.688841,747,60.994818
+3752.208929,1000,61.877345
+4123.596886,1000,62.688987
+3882.418357,1000,63.506384
+1222.403906,319,63.858902
+3950.7844,1000,64.969897
+1133.663277,293,65.222867
+3519.217481,883,65.989581
+3904.164755,1000,67.061183
+3941.813501,1000,67.93606
+3988.277027,1000,68.702283
+3984.765203,1000,69.418604
+2032.9305,518,69.859672
+3925.627003,1000,70.572686
+4121.186448,1000,71.259362
+927.845323,250,71.430974
+681.223494,196,71.570283
+4007.316809,964,72.243597
+3336.355782,808,72.80806
+3918.670195,1000,73.496766
+4111.072906,1000,74.187157
+822.249112,228,74.340477
+4045.926394,1000,75.008166
+3834.889553,931,75.639294
+3770.506637,913,76.272989
+3972.335928,1000,76.969956
+3189.13609,780,77.504433
+4004.26013,998,78.16623
+3952.338062,1000,78.869351
+3864.414001,1000,79.567218
+3234.212861,807,80.140709
+3270.717206,805,80.701072
+3778.118124,1000,81.394286
+3354.853424,819,81.959027
+4066.64844,1000,82.648486
+4032.839659,1000,83.341662
+3633.44696,1000,84.045582
+3976.388662,1000,84.748257
+4107.124104,1000,85.445184
+4015.901456,1000,86.154405
+3858.465243,1000,86.892488
+2531.06318,633,87.350679
+1021.93855,283,87.558814
+3993.111624,1000,88.2503
+3852.176304,951,88.888479
+4047.715459,1000,89.589972
+4037.828587,1000,90.281914
+2699.661937,683,90.773681
+3979.962617,1000,91.484082
+3828.614843,1000,92.175567
+4060.487847,1000,92.85998
+3806.477915,1000,93.559324
+4122.309894,1000,94.244212
+3689.575582,914,94.861114
+3171.393511,792,95.400115
+3697.554232,912,96.013855
+3344.816612,819,96.584449
+3770.726065,906,97.217226
+2758.318688,693,97.699852
+3875.707743,1000,98.388284
+3595.010749,871,98.982749
+3990.355803,1000,99.679666
+3671.843932,1000,100.406731
+3938.453268,1000,101.121792
+3063.812073,745,101.752792
+3665.453826,1000,102.679818
+3172.735876,785,103.280275
+4037.131002,1000,104.504519
+3618.350043,882,105.558064
+4133.87774,1000,106.41744
+3952.157272,1000,107.823448
+3843.714205,1000,108.943133
diff --git a/logs/benchmark/sac-Ant-v3/0.monitor.csv b/logs/benchmark/sac-Ant-v3/0.monitor.csv
new file mode 100644
index 000000000..417ac5065
--- /dev/null
+++ b/logs/benchmark/sac-Ant-v3/0.monitor.csv
@@ -0,0 +1,167 @@
+#{"t_start": 1635519285.5353544, "env_id": "Ant-v3"}
+r,l,t
+5237.897837,1000,1.522428
+5407.915282,1000,2.961156
+5332.542003,1000,4.339916
+5164.29281,1000,5.620398
+5470.146448,1000,6.776858
+5424.560284,1000,7.918268
+4982.68654,1000,9.140854
+5370.92049,1000,10.858078
+3327.076126,1000,12.810981
+5396.412358,1000,14.016499
+1600.10628,324,14.387727
+5261.1635,1000,15.53974
+11.741959,17,15.559266
+5133.014148,1000,16.705739
+5249.246965,1000,17.855005
+5017.616421,1000,18.995355
+5176.247543,1000,20.139415
+5008.37484,1000,21.291844
+5185.332974,1000,22.59589
+5207.121256,1000,24.528967
+5321.60667,1000,26.121284
+5466.515403,1000,27.498967
+4779.800109,1000,28.673935
+1089.807666,233,28.943796
+4156.137171,814,29.935272
+5178.673731,1000,31.121555
+5223.507438,1000,32.576175
+5045.792877,964,33.906193
+5306.700574,1000,35.099559
+1166.435232,233,35.377475
+5230.799915,1000,36.555856
+4876.844494,1000,37.85216
+5392.1342,1000,39.191165
+5215.123662,1000,40.413495
+5289.939511,1000,41.834376
+3672.211042,721,42.963654
+4989.074202,1000,44.230638
+4900.397177,1000,45.847443
+5007.953901,1000,47.042737
+5136.58035,1000,48.264946
+5227.769467,1000,49.421729
+5233.948698,1000,50.702968
+5265.767707,1000,52.013031
+5100.86996,1000,53.283752
+5201.77703,1000,54.521625
+4971.364346,1000,55.678328
+4922.165634,1000,56.840316
+5479.081856,1000,57.944471
+5147.517859,1000,59.122217
+5285.767393,1000,60.369925
+4199.165491,810,61.381308
+2377.7452,1000,63.343206
+290.560087,73,63.573558
+5144.378933,1000,66.019467
+838.245941,212,66.42251
+5075.259479,1000,68.520645
+5301.426022,1000,70.312004
+5409.137861,1000,71.832314
+2290.509022,1000,73.265013
+5229.57814,1000,74.722351
+794.24334,160,74.922002
+4925.713467,1000,76.292198
+5334.554405,1000,78.168507
+4907.917425,1000,79.676369
+4890.954454,1000,80.816692
+5301.590824,1000,81.962854
+5129.413124,1000,83.141782
+5376.229516,1000,84.478671
+214.911619,67,84.558185
+4720.599928,1000,85.761754
+5341.600871,1000,86.957082
+4948.523194,1000,88.149407
+3028.103548,563,88.869065
+5065.762136,1000,90.074296
+2295.04108,475,90.726787
+5119.48135,1000,92.242881
+5186.971678,1000,94.137657
+5134.667195,1000,95.663547
+5283.924467,1000,96.940103
+5277.691851,1000,98.123987
+5549.716975,1000,99.462447
+5093.015028,1000,100.753739
+5080.856519,1000,102.214754
+5366.890113,1000,103.600516
+5183.450586,1000,104.854179
+4845.864855,1000,106.136914
+5335.500687,1000,107.395323
+5262.181426,1000,108.622316
+5392.26651,1000,109.829524
+5155.275331,1000,111.022595
+387.428315,95,111.136172
+5235.954401,1000,112.310759
+5361.875892,1000,113.442432
+5298.993331,1000,114.578672
+5210.236346,1000,115.710679
+5312.231709,1000,116.853714
+5260.122869,1000,117.984629
+2550.874624,508,118.593469
+5470.438871,1000,119.858394
+5390.243685,1000,121.265842
+5271.889492,1000,122.495974
+5382.613746,1000,123.854155
+5186.855912,1000,125.155259
+5205.726423,1000,126.446972
+1513.727561,325,126.932907
+5397.119274,1000,129.21662
+5134.380797,1000,130.651736
+5027.419008,1000,131.862456
+1879.851109,370,132.298047
+5153.349817,1000,133.462209
+5191.633772,1000,134.614284
+5232.869825,1000,135.75106
+4432.341966,856,136.746318
+5231.118783,1000,137.896141
+5235.458696,1000,139.053321
+5249.683478,1000,140.201925
+5164.863329,1000,141.388269
+1730.906055,339,141.790173
+5237.282083,1000,142.989642
+5143.356119,1000,144.569854
+5307.243988,1000,146.989343
+5306.74203,1000,148.704755
+5336.069177,1000,150.210141
+5075.096769,1000,151.771733
+5049.929088,1000,153.204633
+1823.109832,365,153.716839
+5481.331126,1000,155.162368
+5366.954536,1000,156.481486
+5216.339567,1000,157.752029
+1529.297017,1000,159.091492
+3837.136585,707,159.978837
+5145.224474,1000,161.253493
+3197.949567,596,162.007122
+5291.760876,1000,163.358383
+5013.830791,1000,165.169931
+51.581633,32,165.248726
+5153.608517,1000,167.044727
+4443.817998,864,168.208387
+5021.620246,1000,169.514471
+5211.993974,1000,170.716023
+5252.515023,1000,171.965056
+5079.596669,1000,173.302401
+5113.532023,1000,174.799258
+5131.131077,1000,176.212122
+4862.085015,955,177.596722
+5263.660656,1000,178.928786
+5369.378593,1000,180.3507
+1231.993117,245,180.709556
+5535.021855,1000,182.203916
+5232.276251,1000,183.721573
+1829.879056,370,184.316515
+5446.809135,1000,185.707505
+4330.128528,822,186.793282
+5257.352707,1000,188.039451
+3118.521211,1000,189.35889
+4921.4123,959,190.610903
+5306.78472,1000,192.303714
+5190.660661,1000,194.312404
+5278.705728,1000,195.621356
+5068.472347,1000,196.820383
+5420.529634,1000,197.955407
+5126.599483,1000,199.103091
+5407.480759,1000,200.476454
+5243.101982,1000,201.896632
+5301.073823,1000,203.303535
diff --git a/logs/benchmark/sac-HalfCheetah-v3/0.monitor.csv b/logs/benchmark/sac-HalfCheetah-v3/0.monitor.csv
new file mode 100644
index 000000000..da536bf55
--- /dev/null
+++ b/logs/benchmark/sac-HalfCheetah-v3/0.monitor.csv
@@ -0,0 +1,152 @@
+#{"t_start": 1635519613.2402644, "env_id": "HalfCheetah-v3"}
+r,l,t
+9328.225997,1000,0.667051
+9367.045267,1000,1.330729
+9435.059162,1000,2.033533
+9665.814158,1000,2.716248
+9405.683214,1000,3.351512
+9510.280566,1000,4.034523
+9668.81966,1000,4.991978
+9657.166793,1000,6.093707
+9491.285998,1000,6.82705
+9752.444025,1000,7.549113
+9646.404833,1000,8.280412
+9718.333232,1000,8.974798
+9469.180385,1000,9.64991
+9697.178679,1000,10.340975
+9686.625797,1000,11.015214
+9512.572633,1000,11.681941
+9677.591464,1000,12.356046
+9369.437159,1000,13.010681
+9542.299291,1000,13.698387
+9704.998184,1000,14.379847
+9359.990841,1000,15.030694
+9627.465181,1000,15.673186
+9399.278166,1000,16.320058
+9561.827107,1000,16.96469
+9562.942646,1000,17.613035
+9398.031707,1000,18.243327
+9452.341807,1000,18.867104
+9407.878815,1000,19.510252
+9510.264271,1000,20.172288
+9572.825042,1000,20.967423
+9520.613061,1000,21.739523
+9386.255215,1000,22.436643
+9417.053175,1000,23.239609
+9617.098419,1000,23.974434
+9687.154406,1000,24.781559
+9541.946579,1000,25.49113
+9536.91117,1000,26.244645
+9559.94961,1000,26.926429
+9681.57364,1000,27.623954
+9712.204177,1000,28.425619
+9315.56299,1000,29.164805
+9374.589392,1000,29.843147
+9580.065761,1000,30.476086
+9628.610983,1000,31.108121
+9578.084768,1000,31.905385
+9444.467522,1000,32.699818
+9480.061963,1000,33.394359
+9547.978459,1000,34.128607
+9608.089491,1000,34.788953
+9545.426549,1000,35.451637
+9439.918165,1000,36.126348
+9512.124813,1000,36.780804
+9533.366103,1000,37.422119
+9454.079561,1000,38.107479
+9719.948681,1000,38.841504
+9646.108526,1000,39.499264
+9507.64493,1000,40.232132
+9474.856338,1000,41.005003
+9504.173832,1000,41.808694
+9486.930135,1000,42.550869
+9439.137156,1000,43.294301
+9565.323836,1000,44.03491
+9410.101937,1000,44.742364
+9433.791952,1000,45.504613
+9608.703934,1000,46.272998
+9554.260819,1000,47.259565
+9557.581806,1000,48.380897
+9392.428441,1000,49.216309
+9609.780095,1000,49.954806
+9484.018868,1000,50.736311
+9507.469901,1000,51.40662
+9468.697794,1000,52.047121
+9455.544995,1000,52.774742
+9494.236851,1000,53.523197
+9503.411031,1000,54.197982
+9465.546836,1000,54.905735
+9512.36708,1000,55.596461
+9556.633147,1000,56.242114
+9370.860151,1000,56.88258
+9601.262988,1000,57.519803
+9766.783326,1000,58.135993
+9591.745947,1000,58.762163
+9427.267015,1000,59.387379
+9696.500101,1000,60.025112
+9514.451265,1000,60.641712
+9531.435638,1000,61.296114
+9520.767872,1000,62.104377
+9625.173273,1000,63.099225
+9513.973484,1000,64.316943
+9469.642217,1000,65.100842
+9493.480827,1000,65.836246
+9549.39337,1000,66.507228
+9510.342395,1000,67.284941
+9386.798485,1000,67.946287
+9416.998285,1000,68.590995
+9532.944801,1000,69.234617
+9338.789842,1000,69.87721
+9499.110027,1000,70.516711
+9613.086572,1000,71.174834
+9546.042252,1000,71.819252
+9524.862176,1000,72.509275
+9454.331963,1000,73.171765
+9662.241259,1000,73.813837
+9423.700939,1000,74.452663
+9610.802574,1000,75.187854
+9525.050365,1000,75.855475
+9545.66418,1000,76.486804
+9565.711147,1000,77.122037
+9512.090688,1000,77.759949
+9677.746862,1000,78.400348
+9626.735739,1000,79.089928
+9570.536138,1000,79.819335
+9402.267654,1000,80.464915
+9497.266008,1000,81.107611
+9565.138534,1000,81.781866
+9409.963925,1000,82.499945
+9522.557176,1000,83.436192
+9553.947737,1000,84.593942
+9592.565267,1000,85.529606
+9436.157041,1000,86.609833
+9555.817105,1000,87.496722
+9487.809986,1000,88.264773
+9731.733824,1000,89.045524
+9731.215653,1000,89.847692
+9553.241426,1000,90.552973
+9442.202783,1000,91.20897
+9567.045676,1000,91.905877
+9693.146509,1000,92.544946
+9516.019217,1000,93.283144
+9473.531418,1000,93.99945
+9535.459749,1000,94.94383
+9455.304316,1000,96.466333
+9463.621202,1000,97.812074
+9606.217267,1000,98.902821
+9525.966585,1000,99.825855
+9507.847442,1000,100.705936
+9441.743752,1000,101.60551
+9484.267341,1000,102.450822
+9558.174283,1000,103.35931
+9656.370874,1000,104.366483
+9709.369222,1000,105.17136
+9390.91591,1000,106.112692
+9776.238969,1000,107.131109
+9641.880972,1000,108.204873
+9548.312612,1000,109.156781
+9553.307961,1000,110.051115
+9471.257279,1000,110.751783
+9642.012773,1000,111.404264
+9616.996696,1000,112.114126
+9563.296628,1000,113.074755
diff --git a/logs/benchmark/sac-Hopper-v3/0.monitor.csv b/logs/benchmark/sac-Hopper-v3/0.monitor.csv
new file mode 100644
index 000000000..a04b72ae0
--- /dev/null
+++ b/logs/benchmark/sac-Hopper-v3/0.monitor.csv
@@ -0,0 +1,238 @@
+#{"t_start": 1635519881.4798832, "env_id": "Hopper-v3"}
+r,l,t
+1230.760681,329,0.286989
+3618.590938,1000,1.085493
+1252.493582,333,1.350885
+1794.970067,473,1.731241
+3631.648483,1000,2.570779
+3607.968572,1000,3.51929
+3639.235232,1000,4.427158
+1473.283728,392,4.841122
+1240.802264,331,5.197024
+3611.18665,1000,6.330129
+1267.614886,335,6.672603
+3597.126052,1000,8.043199
+3623.894541,1000,8.872503
+1243.726998,330,9.132984
+1367.533824,363,9.415545
+1161.452735,313,9.672043
+3589.377052,1000,10.455236
+1311.236029,346,10.715602
+1229.17542,328,10.967926
+3621.104874,1000,11.901452
+1407.78419,373,12.228494
+1389.920042,368,12.537897
+1459.617108,386,12.863185
+1267.488209,337,13.142159
+1806.720682,477,13.557497
+3662.744671,1000,14.46404
+1202.621893,322,14.768981
+3603.163973,1000,15.735448
+1372.98397,365,16.15349
+3605.182068,1000,17.228932
+3613.32529,1000,18.490822
+1472.034698,392,18.830699
+3617.068031,1000,19.62264
+1277.123074,338,19.881646
+3610.573092,1000,20.65114
+3624.930858,1000,21.4673
+1411.499126,376,21.815776
+3624.683176,1000,22.622191
+1360.134422,360,22.901561
+1182.63762,317,23.148827
+1363.330472,362,23.430365
+1814.090087,480,23.811772
+1229.593252,327,24.056292
+3617.91376,1000,24.80429
+1234.614368,328,25.052755
+1480.346756,396,25.354322
+1449.492556,384,25.652352
+3623.381364,1000,26.503225
+1349.460226,357,26.986883
+3604.124762,1000,28.234132
+1173.73613,316,28.61015
+1416.54453,375,29.044937
+3600.92755,1000,30.489441
+3630.736618,1000,31.756962
+3596.331149,1000,32.714908
+3618.101382,1000,33.546695
+1197.403325,321,33.80641
+3606.918849,1000,34.641803
+3628.387076,1000,35.533974
+3589.315863,1000,36.503783
+3626.554701,1000,37.409305
+1269.914731,335,37.730217
+3606.58172,1000,38.703989
+1401.080864,371,39.06453
+3623.32822,1000,40.019228
+1133.657144,306,40.326061
+3649.599452,1000,41.505299
+1304.593587,344,41.93537
+3595.077124,1000,43.032393
+1472.427401,393,43.613752
+1338.321248,354,44.091938
+3627.33438,1000,45.017639
+3645.351517,1000,45.972415
+1451.938641,383,46.285782
+1352.53154,358,46.600718
+1314.636234,349,46.881286
+1359.725278,360,47.15974
+1444.0621,384,47.45963
+3622.103181,1000,48.26428
+3609.93995,1000,49.162864
+3599.131241,1000,49.981903
+3626.339153,1000,50.764891
+1394.579314,370,51.053033
+1582.41451,419,51.370594
+3621.972419,1000,52.127909
+3608.050584,1000,52.896463
+1264.174344,336,53.146727
+3604.77931,1000,53.91513
+1397.530502,370,54.198092
+3647.54671,1000,54.948042
+1276.33869,337,55.213246
+3618.924019,1000,56.009796
+1386.424862,368,56.297744
+1524.271211,405,56.682899
+1256.518519,333,57.030682
+1322.623598,349,57.339354
+1335.629743,353,57.682005
+1465.451624,387,58.012896
+1419.040242,377,58.337738
+1508.436046,403,58.698526
+3622.050765,1000,59.521648
+1254.844487,332,59.778507
+1285.817128,339,60.034988
+3617.284921,1000,60.881248
+1245.25128,330,61.165899
+3605.326739,1000,61.991446
+3594.059084,1000,62.783978
+1386.332854,367,63.07415
+3615.116359,1000,63.835613
+1386.335815,367,64.125613
+1402.465428,372,64.430521
+1525.407529,407,64.776784
+1377.362836,365,65.074617
+1471.430867,392,65.392884
+1297.019111,343,65.685759
+3614.148581,1000,66.484147
+3634.468136,1000,67.334041
+3626.865138,1000,68.242848
+3619.072732,1000,69.355725
+3620.68169,1000,70.497415
+1210.826233,324,70.785041
+3615.62155,1000,71.717555
+1252.126706,331,71.98991
+3594.353178,1000,72.835696
+1407.593022,374,73.228639
+1411.145635,376,73.742176
+1436.221378,381,74.158049
+3615.108262,1000,75.068415
+1204.458576,322,75.33012
+1288.705905,340,75.596224
+1463.44507,386,75.895713
+1485.50414,392,76.201088
+3632.024479,1000,76.985169
+1500.063438,399,77.303445
+3598.189266,1000,78.08448
+1188.717438,319,78.332211
+1153.035853,310,78.575761
+1321.755542,349,78.846668
+3619.079343,1000,79.628916
+1347.029849,357,79.906839
+3626.131248,1000,80.746564
+1548.651319,411,81.090103
+1397.535012,370,81.393564
+1272.137451,336,81.695815
+1483.089001,396,82.133832
+1319.904891,349,82.532686
+1231.279036,328,82.862131
+1164.755347,314,83.162001
+1472.692959,389,83.598257
+3620.666437,1000,84.692355
+3590.532615,1000,85.745467
+3625.853667,1000,86.79876
+1247.930642,332,87.198553
+1361.073616,361,87.647657
+3635.377862,1000,89.010604
+1137.254343,308,89.449609
+1178.72828,317,89.859264
+1150.507654,310,90.277576
+1348.94722,358,90.776265
+3589.193939,1000,91.750317
+1475.176541,394,92.210593
+1305.788183,344,92.6112
+1155.552433,312,93.148993
+3589.730538,1000,94.833279
+1249.066396,334,95.554368
+1329.519271,351,96.176136
+1352.689933,359,96.697977
+3627.613535,1000,97.899104
+3621.993238,1000,99.177627
+3601.184769,1000,100.258586
+3611.855929,1000,101.172681
+3629.899566,1000,101.980145
+3635.686295,1000,102.976212
+3618.323709,1000,103.872828
+1337.724296,354,104.158272
+1433.248935,381,104.45726
+1222.075661,326,104.806745
+3588.375609,1000,105.752203
+3609.994838,1000,106.648167
+3609.373591,1000,107.478468
+1255.221782,332,107.739221
+1271.81307,338,108.027428
+1481.232356,395,108.394884
+1167.021924,314,108.668529
+1404.628341,373,108.991357
+3615.131766,1000,109.965684
+1275.835732,339,110.305298
+1472.627669,388,110.710991
+3612.78582,1000,111.56852
+3592.765918,1000,112.368391
+3594.03506,1000,113.157425
+3625.803743,1000,113.940287
+1158.124002,311,114.241909
+1337.558239,354,114.5932
+1235.822443,328,114.888763
+3618.777655,1000,115.726878
+3612.347186,1000,116.583133
+1342.494242,354,116.86642
+3613.556736,1000,117.856095
+3585.864316,1000,118.755201
+1216.116843,325,119.078496
+1245.0915,330,119.383898
+3598.898758,1000,120.22449
+1160.020578,312,120.480548
+3629.237384,1000,121.287777
+3607.279866,1000,122.200532
+3663.048568,1000,123.074041
+1320.953115,349,123.348857
+1340.674113,358,123.635249
+1466.672684,387,123.959574
+1413.480168,376,124.269369
+3595.991311,1000,125.065576
+1503.054954,400,125.401912
+3614.380999,1000,126.414685
+3617.21956,1000,127.451318
+1320.486027,349,127.752622
+1294.617289,342,128.0696
+3613.309512,1000,129.141608
+3622.464267,1000,130.039311
+3608.957759,1000,130.894158
+3622.453045,1000,131.879122
+3610.645696,1000,132.711251
+1303.755284,344,133.017312
+3601.709822,1000,133.912844
+1371.414794,363,134.224929
+1404.766967,373,134.609538
+3623.732094,1000,135.458072
+1291.762138,342,135.72987
+3627.25227,1000,136.52969
+3607.462765,1000,137.295227
+1152.020209,310,137.540297
+1161.461235,312,137.787194
+1187.405774,319,138.046302
+1361.391656,361,138.347085
+3585.166729,1000,139.320869
+1818.810275,481,139.753564
diff --git a/logs/benchmark/sac-Humanoid-v3/0.monitor.csv b/logs/benchmark/sac-Humanoid-v3/0.monitor.csv
new file mode 100644
index 000000000..fa34996b1
--- /dev/null
+++ b/logs/benchmark/sac-Humanoid-v3/0.monitor.csv
@@ -0,0 +1,152 @@
+#{"t_start": 1636991795.6490188, "env_id": "Humanoid-v3"}
+r,l,t
+6295.09687,1000,3.369354
+6301.401738,1000,4.832255
+6224.279795,1000,6.304881
+6302.620573,1000,7.76765
+6243.667534,1000,9.230004
+6250.11748,1000,10.702021
+6245.845353,1000,12.170553
+6227.323719,1000,13.638702
+6244.097968,1000,15.112911
+6184.800337,1000,16.582306
+6276.192565,1000,18.040684
+6245.778828,1000,19.505147
+6074.196166,1000,20.978927
+6258.559425,1000,22.436347
+6269.177052,1000,23.903438
+6245.35778,1000,25.365435
+6269.753224,1000,26.826526
+2837.130203,460,27.500005
+6311.178678,1000,28.954204
+6232.924084,1000,30.418972
+6294.436792,1000,31.872714
+6201.996043,1000,33.346954
+6300.150558,1000,34.821703
+6283.576332,1000,36.287675
+6263.22935,1000,37.55729
+6282.84254,1000,38.710926
+6258.833905,1000,39.885105
+6325.443551,1000,41.239267
+6283.180029,1000,42.701662
+6230.693774,1000,44.158178
+6262.132942,1000,45.617537
+6263.604017,1000,47.072073
+6277.618452,1000,48.537458
+6263.497334,1000,50.006338
+6274.382374,1000,51.468263
+6237.439982,1000,52.936788
+6297.368161,1000,54.395969
+6212.360342,1000,55.867377
+6260.760693,1000,57.331578
+6244.032272,1000,58.795477
+6254.394649,1000,60.256076
+6234.990087,1000,61.715013
+6264.560429,1000,63.176899
+6266.510516,1000,64.644033
+6282.556701,1000,66.099836
+6220.325963,1000,67.279855
+6219.631051,1000,68.273212
+6263.95304,1000,69.26757
+6271.115607,1000,70.254154
+6219.022661,1000,71.251736
+6252.468514,1000,72.241936
+6243.275672,1000,73.249219
+6237.438439,1000,74.239716
+6240.439441,1000,75.226493
+6266.155304,1000,76.213796
+6263.56421,1000,77.201309
+6237.679988,1000,78.188149
+6195.108655,1000,79.191404
+6287.354711,1000,80.193928
+6244.773964,1000,81.202729
+6196.354731,1000,82.217118
+6243.395239,1000,83.226913
+6284.350364,1000,84.245429
+6268.076784,1000,85.254379
+6230.080629,1000,86.273344
+6279.171218,1000,87.292922
+6258.971314,1000,88.295994
+6264.993408,1000,89.280228
+6258.420129,1000,90.268801
+6278.321859,1000,91.256935
+6281.897649,1000,92.240784
+6271.720302,1000,93.249151
+6219.869006,1000,94.254185
+6269.133224,1000,95.238871
+6225.442091,1000,96.23377
+6234.56855,1000,97.251906
+6238.596454,1000,98.259493
+6279.482728,1000,99.265662
+6281.42632,1000,100.252428
+6259.958172,1000,101.239922
+6306.670028,1000,102.230647
+6252.917864,1000,103.219375
+6244.186566,1000,104.241529
+6243.723745,1000,105.284219
+6201.915556,1000,106.282625
+6234.028761,1000,107.319313
+6233.308046,1000,108.374446
+6238.86908,1000,109.365787
+6299.99493,1000,110.393233
+6237.318385,1000,111.831067
+6201.995802,1000,113.268844
+6268.880349,1000,114.69103
+6280.167265,1000,116.11859
+6280.743893,1000,117.53564
+6246.288479,1000,118.961478
+6262.702614,1000,120.396389
+6213.907163,1000,121.838295
+6230.110475,1000,123.278828
+6227.542872,1000,124.709319
+6266.478248,1000,126.140775
+6223.342381,1000,127.575513
+6278.992975,1000,129.002483
+6277.42317,1000,130.43641
+6285.56072,1000,131.85823
+6269.882111,1000,133.284563
+6298.016492,1000,134.707321
+6290.890273,1000,136.121903
+6291.651742,1000,137.543156
+6243.046572,1000,138.973706
+6284.591343,1000,140.40161
+6236.500348,1000,141.818691
+6247.652592,1000,143.240197
+6236.270762,1000,144.664375
+6249.000394,1000,146.093176
+6265.357019,1000,147.421951
+6246.09252,1000,148.569778
+6245.196317,1000,149.710508
+6238.987303,1000,151.112863
+6221.350304,1000,152.538789
+6290.946656,1000,153.968976
+6259.256853,1000,155.397217
+6224.04731,1000,156.841569
+6291.766758,1000,158.27088
+6268.076646,1000,159.686794
+6210.908719,1000,161.12275
+6277.994421,1000,162.540398
+6289.838942,1000,163.95885
+6276.479635,1000,165.373176
+6278.520364,1000,166.794743
+6189.807303,1000,168.221501
+6260.101844,1000,169.643124
+6243.118407,1000,171.072263
+6245.910167,1000,172.502799
+6244.166456,1000,173.928475
+6251.494792,1000,175.364047
+6287.555953,1000,176.783449
+6270.597195,1000,178.20414
+6260.749403,1000,179.627964
+6249.728383,1000,181.057173
+6237.938458,1000,182.491023
+6274.644585,1000,183.913974
+6284.782008,1000,185.335639
+6202.486415,1000,186.766998
+6261.366723,1000,188.192742
+6310.520981,1000,189.61082
+6260.943494,1000,191.026066
+6230.407171,1000,192.459632
+6235.276468,1000,193.891362
+6290.841604,1000,195.336529
+6234.661845,1000,196.789763
diff --git a/logs/benchmark/sac-Swimmer-v3/0.monitor.csv b/logs/benchmark/sac-Swimmer-v3/0.monitor.csv
new file mode 100644
index 000000000..c0a6c825b
--- /dev/null
+++ b/logs/benchmark/sac-Swimmer-v3/0.monitor.csv
@@ -0,0 +1,152 @@
+#{"t_start": 1635519493.7939534, "env_id": "Swimmer-v3"}
+r,l,t
+345.74961,1000,0.693904
+344.052901,1000,1.36459
+349.749145,1000,2.032438
+345.583676,1000,2.74248
+340.006668,1000,3.718818
+341.760126,1000,5.028484
+347.716905,1000,6.382917
+342.189052,1000,7.447895
+343.356554,1000,8.448687
+349.302776,1000,9.521941
+344.361904,1000,10.379532
+346.990959,1000,11.175025
+346.009075,1000,11.966118
+340.831426,1000,12.757102
+348.305924,1000,13.419221
+341.645044,1000,14.089081
+349.678197,1000,14.765045
+338.526885,1000,15.437658
+340.547477,1000,16.132973
+346.672105,1000,16.941021
+349.202495,1000,17.690562
+343.896936,1000,18.464039
+343.963497,1000,19.167107
+348.953076,1000,19.87532
+346.823179,1000,20.598723
+348.841025,1000,21.320855
+345.728473,1000,22.133799
+344.637173,1000,23.296785
+340.013975,1000,24.378085
+347.931886,1000,25.708586
+344.023671,1000,26.797019
+346.471979,1000,27.744511
+349.20225,1000,28.76128
+347.298162,1000,29.868746
+348.132835,1000,30.803827
+346.730733,1000,31.688959
+340.126725,1000,32.734888
+344.900894,1000,33.66311
+342.124378,1000,34.534899
+349.140855,1000,35.294881
+345.343734,1000,36.022261
+348.251556,1000,36.70738
+345.225802,1000,37.37588
+348.691493,1000,38.076038
+350.161667,1000,38.985168
+344.295233,1000,39.922438
+346.061377,1000,40.745001
+348.049659,1000,41.541267
+345.013851,1000,42.26512
+346.898277,1000,42.952546
+348.303932,1000,43.632929
+339.706612,1000,44.384531
+344.95803,1000,45.104799
+342.824472,1000,45.808528
+345.402733,1000,46.501819
+342.424988,1000,47.186894
+348.131542,1000,47.866642
+344.877033,1000,48.700326
+347.024779,1000,49.555814
+349.794982,1000,50.35517
+341.21619,1000,51.045394
+342.71452,1000,51.702816
+348.979969,1000,52.511201
+342.551771,1000,53.380832
+346.546376,1000,54.174605
+340.475186,1000,54.857372
+344.40853,1000,55.644693
+341.461542,1000,56.32673
+343.728021,1000,57.001884
+347.363697,1000,57.665542
+349.827506,1000,58.37839
+343.522209,1000,59.126513
+349.102854,1000,59.919763
+338.366813,1000,60.617201
+345.382275,1000,61.319354
+339.764556,1000,62.076783
+349.303181,1000,62.774482
+339.446642,1000,63.478946
+347.669785,1000,64.161578
+348.118894,1000,64.832659
+348.062283,1000,65.65208
+338.482444,1000,66.408782
+349.34305,1000,67.10768
+345.428719,1000,67.88109
+346.956801,1000,68.601475
+346.631446,1000,69.27774
+345.418611,1000,69.959245
+346.973555,1000,70.685372
+347.193149,1000,71.375602
+349.064105,1000,72.071609
+347.482163,1000,72.741375
+346.917947,1000,73.40228
+346.680542,1000,74.186963
+343.695292,1000,74.915671
+345.91094,1000,75.642312
+339.374772,1000,76.38685
+341.545736,1000,77.077276
+347.436834,1000,77.768198
+341.180425,1000,78.515412
+348.269916,1000,79.223594
+343.885375,1000,80.008315
+349.559843,1000,80.743209
+347.932835,1000,81.495442
+347.338551,1000,82.193586
+346.724765,1000,82.895764
+341.54217,1000,83.683337
+349.469982,1000,84.396433
+343.276633,1000,85.072628
+347.251779,1000,85.857149
+348.364765,1000,86.585138
+340.729422,1000,87.389802
+348.087243,1000,88.416223
+348.024453,1000,89.75282
+348.542763,1000,90.56546
+343.664869,1000,91.262752
+344.634603,1000,91.947422
+343.515144,1000,92.656897
+345.05472,1000,93.467553
+340.705316,1000,94.185045
+350.216914,1000,94.897869
+344.134253,1000,95.696499
+342.343982,1000,96.421743
+345.182811,1000,97.114929
+345.034001,1000,97.797013
+346.898092,1000,98.459564
+347.866613,1000,99.103626
+341.214989,1000,99.775826
+347.600891,1000,100.46956
+349.342537,1000,101.148812
+347.774622,1000,101.823037
+346.709735,1000,102.490302
+347.347669,1000,103.153809
+348.115526,1000,103.821032
+349.623052,1000,104.490142
+347.72047,1000,105.168986
+348.948977,1000,105.84557
+348.301221,1000,106.561285
+341.464733,1000,107.331498
+342.614249,1000,108.019221
+346.93566,1000,108.760883
+349.603668,1000,109.487535
+338.174643,1000,110.229753
+343.25426,1000,110.950818
+349.588761,1000,111.622911
+345.3636,1000,112.277351
+345.591866,1000,112.98254
+345.81833,1000,113.730223
+344.023522,1000,114.561938
+348.160912,1000,115.319092
+347.372045,1000,115.975665
diff --git a/logs/benchmark/sac-Walker2d-v3/0.monitor.csv b/logs/benchmark/sac-Walker2d-v3/0.monitor.csv
new file mode 100644
index 000000000..8c4f0c040
--- /dev/null
+++ b/logs/benchmark/sac-Walker2d-v3/0.monitor.csv
@@ -0,0 +1,152 @@
+#{"t_start": 1635519730.4202006, "env_id": "Walker2d-v3"}
+r,l,t
+3884.155776,1000,0.984004
+3780.756451,1000,2.001744
+3975.735752,1000,3.029841
+3949.667943,1000,3.921685
+3744.385069,1000,5.100599
+4008.238202,1000,6.149827
+3965.821918,1000,7.171516
+3941.093791,1000,8.200778
+3912.671155,1000,9.218587
+3722.500646,1000,10.213411
+3944.694646,1000,11.397287
+3744.227285,1000,12.360232
+3736.869975,1000,13.295457
+3907.699038,1000,14.307275
+3969.436803,1000,15.243389
+3955.654602,1000,16.19352
+3937.086621,1000,17.131851
+3846.003331,1000,18.055707
+3928.670717,1000,19.202942
+3821.811933,1000,20.343901
+3939.270852,1000,21.373228
+3852.080725,1000,22.374084
+3977.430852,1000,23.371797
+4037.869451,1000,24.441668
+3969.595697,1000,25.425196
+3961.273725,1000,26.463655
+3948.799378,1000,28.050772
+3827.30171,1000,29.320498
+3887.931317,1000,30.189736
+3951.785439,1000,31.117627
+3955.388685,1000,32.012513
+3891.018477,1000,32.873956
+3863.680473,1000,33.654311
+3751.215853,1000,34.476296
+3937.441511,1000,35.375817
+3797.858325,1000,36.188355
+3829.810432,1000,36.986204
+3815.426338,1000,37.805427
+3968.787454,1000,38.675202
+3879.074547,1000,39.573842
+3924.546648,1000,40.847621
+3979.264489,1000,42.692857
+4001.227839,1000,44.461255
+3861.161577,1000,46.072112
+3780.721203,1000,47.137214
+3905.436888,1000,48.136456
+3954.016889,1000,49.075744
+3831.07193,1000,49.96429
+3863.565149,1000,51.077747
+4005.244361,1000,52.10105
+3800.488438,1000,52.995273
+3777.726537,1000,53.880662
+3934.496188,1000,54.973056
+3828.938824,1000,55.953111
+3824.043225,1000,56.865591
+3727.150815,1000,57.841754
+3815.33585,1000,58.801031
+3783.836789,1000,59.649241
+3850.380751,1000,60.539005
+3882.1758,1000,61.381682
+3928.854024,1000,62.559529
+3890.950418,1000,63.888848
+3975.542223,1000,65.294117
+4008.419419,1000,66.214627
+3910.12384,1000,67.057043
+3909.004893,1000,68.008908
+3754.326221,1000,69.215247
+3934.643368,1000,70.352663
+3892.670613,1000,71.332481
+4020.809381,1000,72.180744
+3904.917415,1000,73.064235
+3895.123201,1000,73.930998
+3774.627553,1000,74.74244
+3858.490876,1000,75.800117
+3825.010089,1000,76.981659
+3778.774008,1000,78.556821
+3929.108814,1000,79.60905
+3743.767348,1000,80.522291
+3831.87514,1000,81.379614
+3996.936659,1000,82.225183
+3898.653976,1000,83.068811
+3877.38385,1000,84.075534
+3693.506143,1000,85.010343
+3861.56301,1000,85.924392
+3965.920402,1000,86.740651
+3894.675045,1000,87.560811
+4038.856556,1000,88.361552
+3948.457661,1000,89.174484
+3929.640673,1000,90.146212
+3787.308234,1000,91.064236
+3757.514654,1000,91.933197
+3919.868137,1000,93.132345
+3888.788387,1000,94.113212
+3879.297448,1000,95.087872
+3927.60075,1000,96.026069
+4052.96801,1000,96.958626
+3858.566495,1000,97.863629
+3968.94195,1000,98.893032
+4002.446791,1000,99.886379
+3682.587161,1000,100.745362
+3779.518922,1000,101.576627
+3947.414781,1000,102.538526
+3811.490469,1000,103.400932
+3797.188828,1000,104.211265
+3890.898107,1000,105.265802
+3933.303334,1000,106.785921
+3806.18015,1000,108.044033
+3736.563765,1000,109.012887
+3865.269596,1000,109.825776
+3781.560813,1000,110.600134
+3965.403923,1000,111.391109
+3850.531615,1000,112.16777
+3898.237585,1000,112.986322
+3865.192912,1000,113.867457
+3962.011548,1000,114.743936
+3888.860425,1000,115.597007
+3800.062981,1000,116.511002
+3855.986438,1000,117.408245
+3821.409315,1000,118.282482
+3921.789593,1000,119.166786
+3857.060882,1000,120.058381
+3940.015648,1000,120.993303
+3829.725703,1000,121.896513
+3857.671243,1000,122.778067
+916.840004,321,123.084891
+3984.905019,1000,123.913567
+3782.83237,1000,124.821303
+3863.529953,1000,125.770969
+3972.479594,1000,126.736805
+3826.96793,1000,128.03593
+3740.170997,1000,129.089395
+3912.532168,1000,129.929502
+4016.098595,1000,130.826608
+3844.022173,1000,131.75387
+3939.138368,988,132.647746
+3861.360938,1000,133.599187
+3859.499546,1000,134.43326
+3845.065819,1000,135.2695
+3974.982631,1000,136.171972
+3940.037683,1000,137.42666
+3990.706233,1000,138.907336
+3862.783009,1000,139.939779
+3990.62025,1000,140.819044
+3801.640589,1000,141.682208
+3780.613813,1000,142.530988
+3899.274816,1000,143.431598
+3864.175615,1000,144.327114
+3846.794695,1000,145.22326
+3947.817632,1000,146.101427
+3902.654266,1000,146.965182
diff --git a/logs/benchmark/td3-Ant-v3/0.monitor.csv b/logs/benchmark/td3-Ant-v3/0.monitor.csv
new file mode 100644
index 000000000..9be284c48
--- /dev/null
+++ b/logs/benchmark/td3-Ant-v3/0.monitor.csv
@@ -0,0 +1,153 @@
+#{"t_start": 1635518699.6012485, "env_id": "Ant-v3"}
+r,l,t
+5913.315474,1000,0.729284
+6007.599242,1000,1.396673
+5970.329705,1000,2.140175
+5863.421973,1000,2.949105
+133.968681,45,2.982478
+5761.578538,1000,3.738882
+5906.753748,1000,4.464274
+5782.525809,1000,5.175434
+5764.587252,1000,5.896517
+5897.512863,1000,6.636969
+5899.187239,1000,7.358533
+5965.905151,1000,8.053768
+5889.09236,1000,8.780891
+5750.341603,1000,9.646121
+1833.441282,348,10.099056
+5983.644358,1000,11.252231
+5869.99497,1000,12.386524
+5867.299694,1000,13.715882
+5947.154884,1000,14.855156
+5871.506164,1000,16.038849
+5872.496861,1000,17.41264
+5927.783198,1000,18.825502
+5817.175057,1000,20.276547
+5847.979033,1000,21.372158
+5851.102277,1000,22.36085
+5922.279312,1000,23.33088
+5848.086836,1000,24.322867
+5977.642126,1000,25.31077
+5895.8795,1000,26.28663
+5990.193481,1000,27.25905
+5878.722591,1000,28.312329
+4273.462486,1000,29.545991
+5944.130042,1000,30.629415
+5908.521323,1000,31.830247
+5960.194943,1000,33.055136
+5920.760768,1000,34.34768
+5890.457923,1000,35.356807
+5826.299717,1000,36.41707
+5900.427757,1000,37.490049
+5953.079442,1000,38.480645
+5943.757464,1000,39.485915
+5639.556828,1000,40.502553
+6011.581064,1000,41.493531
+5926.724266,1000,42.476238
+5878.594381,1000,43.432753
+5825.899375,1000,44.397197
+5733.205776,1000,45.365462
+6008.401709,1000,46.340214
+5942.554812,1000,47.34071
+6039.996522,1000,48.315594
+5809.092552,1000,49.288611
+5895.076294,1000,50.250932
+5726.317805,1000,51.214367
+5801.820906,1000,52.184892
+5814.234278,1000,53.147975
+5841.380088,1000,54.112452
+5814.818066,1000,55.072491
+5895.597141,1000,56.047278
+5900.265188,1000,57.011499
+5915.523936,1000,57.976101
+5955.259779,1000,58.956407
+5988.654008,1000,59.916112
+5785.065993,1000,60.879017
+5860.233861,1000,61.844926
+6050.644692,1000,62.825675
+5984.62811,1000,63.932767
+6030.433566,1000,65.216947
+5882.135812,1000,66.522858
+5772.300693,1000,67.67405
+5798.89402,1000,68.932011
+5887.338176,1000,70.14977
+5948.327876,1000,71.210706
+5973.693064,1000,72.515635
+5967.879276,1000,74.306679
+5957.194037,1000,75.960274
+5879.771684,1000,77.516975
+5936.959144,1000,78.879452
+5992.688992,1000,80.132463
+5833.997306,1000,81.179469
+5863.805042,1000,82.221855
+5785.231173,1000,83.245736
+5856.596758,1000,84.311985
+5973.924631,1000,85.378637
+5964.312662,1000,86.400698
+5796.339059,1000,87.487292
+5862.086774,1000,88.596757
+5759.166812,1000,89.592646
+5758.029402,1000,90.569009
+6024.203186,1000,91.550138
+5900.452953,1000,92.536298
+5745.711221,1000,93.529209
+6052.277271,1000,94.572032
+5926.975088,1000,95.582342
+5857.986736,1000,96.584657
+5802.483426,1000,97.719677
+5927.295922,1000,99.005076
+5812.809683,1000,100.380903
+5971.224775,1000,102.04328
+5828.881138,1000,103.183527
+5892.2724,1000,104.339387
+5985.715367,1000,105.460064
+5768.213904,1000,106.616506
+5892.095082,1000,107.758052
+6002.529471,1000,108.794694
+5845.770972,1000,109.824095
+5601.838831,1000,110.804967
+5887.614952,1000,111.79291
+5978.954238,1000,112.869967
+5930.060221,1000,113.99368
+6068.648348,1000,115.068879
+5893.784073,1000,116.057529
+5965.482516,1000,117.067971
+5835.756016,1000,118.06063
+6031.612439,1000,119.041332
+5810.599359,1000,120.02486
+5913.302372,1000,121.011831
+5854.967833,1000,122.011148
+5865.750335,1000,123.060149
+5990.611595,1000,124.017112
+5868.303282,1000,124.970663
+5750.045006,1000,125.924134
+5899.459756,1000,126.912413
+5759.517166,1000,127.878185
+6020.028659,1000,128.848599
+5852.937594,1000,129.828027
+5857.060163,1000,130.808326
+5940.016254,1000,131.80607
+5787.315099,1000,132.784003
+5811.298899,1000,133.969036
+5750.831877,1000,135.091351
+5904.322053,1000,136.128989
+5911.782327,1000,137.175287
+5894.042054,1000,138.180765
+5693.854254,1000,139.159188
+6090.702273,1000,140.27654
+6007.076347,1000,141.288674
+5889.469849,1000,142.450347
+5868.784457,1000,143.620903
+6098.065653,1000,144.753503
+5823.944957,1000,145.831212
+6058.133959,1000,146.938405
+5933.166273,1000,148.097855
+5790.559173,1000,149.104136
+5946.315094,1000,150.057576
+5842.209764,1000,151.007397
+5761.931976,1000,151.965596
+5912.175017,1000,152.921952
+5752.611588,1000,153.878018
+5973.046732,1000,154.858627
+5951.76496,1000,155.818936
+5781.879893,1000,156.832309
diff --git a/logs/benchmark/td3-HalfCheetah-v3/0.monitor.csv b/logs/benchmark/td3-HalfCheetah-v3/0.monitor.csv
new file mode 100644
index 000000000..dcecf3fbf
--- /dev/null
+++ b/logs/benchmark/td3-HalfCheetah-v3/0.monitor.csv
@@ -0,0 +1,152 @@
+#{"t_start": 1635518959.4834206, "env_id": "HalfCheetah-v3"}
+r,l,t
+9786.399943,1000,0.708283
+9862.285544,1000,1.353662
+9928.766232,1000,2.043035
+9986.564834,1000,2.698953
+9816.701068,1000,3.331441
+9757.505751,1000,4.025127
+9830.035016,1000,4.690306
+9971.654049,1000,5.4065
+9823.619884,1000,6.003721
+9734.315053,1000,6.520714
+9813.644303,1000,7.02146
+9592.862736,1000,7.513396
+10208.87398,1000,8.007597
+9888.603106,1000,8.499662
+9662.272436,1000,8.98546
+9668.37115,1000,9.490434
+9651.867493,1000,9.978622
+9854.388414,1000,10.481315
+9473.307687,1000,10.994772
+9699.236007,1000,11.510611
+9961.74864,1000,12.026928
+9683.163786,1000,12.553415
+9751.859354,1000,13.056211
+9617.191806,1000,13.59972
+9388.497173,1000,14.142205
+9805.397767,1000,14.671571
+9925.577887,1000,15.205163
+9873.488693,1000,15.730322
+10028.8013,1000,16.269219
+9793.345058,1000,16.785503
+9982.033629,1000,17.285619
+10184.689513,1000,17.785408
+9693.264346,1000,18.314686
+4689.067306,1000,18.874382
+1577.72697,1000,19.44814
+9795.665969,1000,19.981401
+9918.906854,1000,20.545877
+10000.559426,1000,21.136367
+10158.900873,1000,21.751374
+9823.929755,1000,22.316455
+9884.482114,1000,22.86484
+9696.165023,1000,23.417611
+9962.753106,1000,23.986811
+9727.97491,1000,24.602272
+9575.42984,1000,25.18
+9554.322589,1000,25.729409
+9768.975747,1000,26.295774
+9765.226196,1000,26.839467
+9714.064195,1000,27.397312
+9516.946816,1000,27.969908
+9539.717121,1000,28.546867
+9683.188107,1000,29.130594
+9649.607213,1000,29.685679
+9636.809717,1000,30.215259
+9755.394164,1000,30.71768
+9612.917757,1000,31.290762
+9652.74811,1000,31.864116
+10063.580457,1000,32.415487
+9745.973445,1000,32.998371
+9805.796648,1000,33.61462
+9857.110684,1000,34.144902
+10020.896871,1000,34.664049
+9774.674015,1000,35.184981
+9646.703941,1000,35.686461
+9648.336521,1000,36.206142
+9501.294946,1000,36.708071
+9976.930597,1000,37.212308
+9689.411283,1000,37.708412
+9869.629641,1000,38.22922
+9978.549103,1000,38.735342
+9399.665627,1000,39.27205
+3013.264687,1000,39.790823
+9600.099708,1000,40.366419
+9763.350806,1000,41.041224
+9770.247172,1000,41.750947
+9702.614832,1000,42.320428
+10079.959235,1000,42.93951
+9808.137157,1000,43.652605
+9981.054399,1000,44.392103
+9886.087882,1000,45.01112
+9503.939585,1000,45.545516
+9602.662572,1000,46.066805
+9611.70092,1000,46.627603
+9951.103576,1000,47.33954
+9541.780898,1000,47.940007
+9700.95694,1000,48.558041
+9969.52637,1000,49.104066
+9670.811913,1000,49.638004
+9889.511462,1000,50.29425
+10000.449833,1000,50.921911
+9776.884666,1000,51.465378
+9741.073462,1000,51.997566
+9853.812587,1000,52.514993
+9661.120721,1000,53.01747
+10024.254798,1000,53.535096
+9549.116565,1000,54.04693
+9644.605728,1000,54.630272
+9612.156744,1000,55.167414
+9583.802593,1000,55.684733
+9638.748238,1000,56.200807
+9764.203596,1000,56.71315
+9830.902334,1000,57.223452
+10325.407453,1000,57.747242
+9824.928962,1000,58.255081
+9975.377094,1000,58.840531
+9762.911762,1000,59.510327
+9826.79354,1000,60.072606
+9747.697014,1000,60.658744
+9972.423678,1000,61.250853
+10109.45398,1000,61.980212
+9913.414517,1000,62.6107
+9937.174499,1000,63.204551
+9574.899592,1000,63.727859
+9731.204576,1000,64.238884
+9792.645973,1000,64.770414
+9695.119094,1000,65.307035
+10010.299378,1000,65.826534
+9672.378715,1000,66.339357
+9802.322656,1000,66.85099
+9603.002127,1000,67.42314
+10099.805253,1000,67.969617
+9845.413183,1000,68.489748
+9936.862936,1000,69.012454
+9721.686175,1000,69.566371
+9271.53352,1000,70.171078
+9760.646284,1000,70.767487
+9869.045861,1000,71.368549
+9724.302752,1000,71.963645
+9738.348926,1000,72.50214
+9716.357069,1000,73.168746
+9863.147457,1000,73.718274
+9871.07554,1000,74.24857
+9546.629378,1000,74.767653
+9964.938748,1000,75.338906
+9949.287656,1000,75.921793
+10046.716342,1000,76.62572
+9786.27797,1000,77.341002
+9749.120963,1000,77.93383
+9937.668463,1000,78.507509
+9640.627518,1000,79.052021
+9866.859125,1000,79.639594
+9765.328011,1000,80.237292
+9718.024873,1000,80.827706
+9929.033626,1000,81.449718
+9725.20211,1000,82.033036
+9498.718137,1000,82.627043
+10026.661658,1000,83.212086
+9983.641541,1000,83.804583
+9630.416253,1000,84.345545
+9820.766941,1000,84.879538
diff --git a/logs/benchmark/td3-Hopper-v3/0.monitor.csv b/logs/benchmark/td3-Hopper-v3/0.monitor.csv
new file mode 100644
index 000000000..c26427d33
--- /dev/null
+++ b/logs/benchmark/td3-Hopper-v3/0.monitor.csv
@@ -0,0 +1,152 @@
+#{"t_start": 1635519164.1197133, "env_id": "Hopper-v3"}
+r,l,t
+3603.646445,1000,0.79679
+3603.9284,1000,1.586663
+3608.124928,1000,2.356995
+3599.75101,1000,3.14597
+3607.279312,1000,3.837293
+3609.034194,1000,4.603998
+3610.498524,1000,5.284259
+3607.085762,1000,5.989361
+3608.883446,1000,6.719693
+3603.261528,1000,7.437218
+3608.189465,1000,8.109525
+3608.707638,1000,8.77759
+3604.65967,1000,9.441556
+3604.139167,1000,10.1305
+3608.174005,1000,10.817646
+3605.4388,1000,11.507439
+3610.676637,1000,12.189455
+3608.714769,1000,12.869707
+3609.609007,1000,13.597299
+3609.717259,1000,14.286545
+3605.177585,1000,14.969407
+3605.874556,1000,15.646764
+3605.557032,1000,16.327515
+3610.934301,1000,16.999189
+3607.673996,1000,17.699639
+3607.73486,1000,18.375724
+3606.182636,1000,19.059695
+3608.264075,1000,19.789908
+3599.035516,1000,20.457277
+3609.507487,1000,21.11121
+3606.801745,1000,21.816272
+3597.968287,1000,22.546178
+3605.623085,1000,23.220157
+3610.064306,1000,23.858231
+3607.021332,1000,24.494883
+3599.126522,1000,25.131965
+3601.883127,1000,25.773662
+3598.557274,1000,26.411253
+3609.353761,1000,27.118974
+3600.110194,1000,27.787088
+3601.927443,1000,28.440077
+3610.742097,1000,29.092243
+3607.11591,1000,29.776118
+3609.788711,1000,30.49671
+3610.383246,1000,31.356987
+3605.815209,1000,32.146479
+3599.330296,1000,32.868699
+3602.404164,1000,33.716874
+3591.35391,1000,34.938531
+3599.132031,1000,35.909016
+3606.019907,1000,36.85623
+3609.363343,1000,37.950608
+3600.114269,1000,38.810988
+3606.971947,1000,39.575834
+3606.84142,1000,40.56781
+3607.678856,1000,41.484448
+3609.364274,1000,42.264345
+3602.866106,1000,42.998072
+3608.966823,1000,44.006913
+3607.340982,1000,44.925455
+3603.20902,1000,45.650321
+3609.335816,1000,46.358207
+3603.310634,1000,47.049314
+3612.665911,1000,47.733932
+3607.627442,1000,48.464282
+3602.513019,1000,49.164521
+3606.128654,1000,50.04873
+3608.897412,1000,51.048141
+3607.650681,1000,51.800174
+3607.583858,1000,52.51961
+3608.393387,1000,53.378908
+3603.414093,1000,54.313754
+3598.57946,1000,55.161221
+3606.574668,1000,55.954219
+3607.700773,1000,56.869976
+3610.340119,1000,57.735712
+3608.107103,1000,58.619683
+3602.961677,1000,59.512862
+3604.503324,1000,60.371811
+3601.630892,1000,61.406693
+3600.808096,1000,62.720903
+3610.963998,1000,64.040542
+3609.428842,1000,65.214006
+3608.195512,1000,66.46628
+3606.731035,1000,67.801288
+3608.665484,1000,68.937464
+3608.904207,1000,69.701859
+3602.046617,1000,70.377155
+3610.805774,1000,71.373108
+3604.206874,1000,72.295954
+3600.918097,1000,73.018557
+3610.099008,1000,73.957025
+3604.71074,1000,74.881347
+3610.738606,1000,75.571893
+3610.64653,1000,76.222234
+3610.036043,1000,76.871573
+3605.80935,1000,77.527059
+3601.461108,1000,78.248617
+3607.582092,1000,78.920456
+3607.565172,1000,79.610035
+3608.89086,1000,80.312667
+3607.005492,1000,80.990374
+3600.00827,1000,81.690001
+3603.830941,1000,82.409552
+3612.576522,1000,83.385608
+3607.010022,1000,84.187812
+3594.260843,1000,84.856401
+3605.949951,1000,85.502709
+3607.703959,1000,86.144963
+3605.552696,1000,86.788846
+3609.369507,1000,87.420601
+3603.642268,1000,88.057372
+3609.684774,1000,88.694375
+3606.452024,1000,89.337953
+3613.798465,1000,89.974725
+3606.944945,1000,90.610304
+3608.491344,1000,91.246065
+3604.419022,1000,91.886069
+3612.23498,1000,92.533866
+3601.009457,1000,93.377409
+3614.139085,1000,94.206789
+3606.46939,1000,94.981045
+3610.561423,1000,95.758513
+3603.44362,1000,96.591897
+3612.776328,1000,97.388014
+3607.831118,1000,98.115208
+3605.517963,1000,98.866582
+3600.903108,1000,99.637663
+3607.994112,1000,100.357065
+3610.528249,1000,101.097984
+3607.695741,1000,101.805622
+3612.730359,1000,102.525701
+3604.836548,1000,103.177749
+3595.965386,1000,103.815218
+3611.075753,1000,104.55627
+3608.931629,1000,105.270046
+3611.838955,1000,105.945259
+3603.058602,1000,106.639047
+3605.526072,1000,107.57097
+3611.363223,1000,108.713326
+3604.025757,1000,109.907731
+3608.192326,1000,110.739065
+3606.387515,1000,111.467974
+3610.416706,1000,112.251598
+3611.481547,1000,112.945488
+3601.224428,1000,113.63365
+3602.090189,1000,114.37353
+3610.884789,1000,115.287493
+3606.992824,1000,116.50632
+3607.793409,1000,117.617262
diff --git a/logs/benchmark/td3-Humanoid-v3/0.monitor.csv b/logs/benchmark/td3-Humanoid-v3/0.monitor.csv
new file mode 100644
index 000000000..05b391abc
--- /dev/null
+++ b/logs/benchmark/td3-Humanoid-v3/0.monitor.csv
@@ -0,0 +1,152 @@
+#{"t_start": 1636991638.937492, "env_id": "Humanoid-v3"}
+r,l,t
+5564.729567,1000,2.952954
+5557.627895,1000,3.987336
+5583.778507,1000,5.020298
+5563.750653,1000,6.058141
+5557.435189,1000,7.104638
+5552.869884,1000,8.151777
+5566.59141,1000,9.201447
+5551.119477,1000,10.247021
+5549.494999,1000,11.295877
+5544.105637,1000,12.343779
+5580.553812,1000,13.38937
+5576.020272,1000,14.376231
+5576.258996,1000,15.248868
+5568.945709,1000,16.107316
+5571.068852,1000,16.974191
+5554.181224,1000,17.843382
+5569.724637,1000,18.714371
+5561.105907,1000,19.581948
+5584.571549,1000,20.447118
+5579.428427,1000,21.311344
+5576.62736,1000,22.177701
+5580.701043,1000,23.041018
+5595.995233,1000,23.902326
+5588.749372,1000,24.764616
+5560.93593,1000,25.635088
+5567.447402,1000,26.505061
+5607.948919,1000,27.36731
+5578.027084,1000,28.236561
+5569.309213,1000,29.101407
+5560.805454,1000,29.985653
+5579.552763,1000,30.888655
+5576.457797,1000,31.785273
+5570.437933,1000,32.723468
+5586.483115,1000,33.620865
+5569.312111,1000,34.510183
+5558.09901,1000,35.515446
+5575.371101,1000,36.560475
+5560.696276,1000,37.605896
+5586.675166,1000,38.64876
+5537.11207,1000,39.693221
+5574.206035,1000,40.737603
+5536.386393,1000,41.787312
+5574.956135,1000,42.831958
+5560.709902,1000,43.873917
+5551.783404,1000,44.921608
+5562.405921,1000,45.965423
+5534.007238,1000,47.019906
+5540.981498,1000,48.06456
+5567.97003,1000,49.110852
+5547.021339,1000,50.157464
+5563.963385,1000,51.207851
+5541.333693,1000,52.267116
+5557.700028,1000,53.318619
+5568.590369,1000,54.365649
+5587.482478,1000,55.41464
+5573.167749,1000,56.465482
+5558.789851,1000,57.513237
+5562.632168,1000,58.559676
+5561.823959,1000,59.602617
+5579.103469,1000,60.645841
+5548.195515,1000,61.707599
+5566.526275,1000,62.753356
+5559.062355,1000,63.807394
+5559.998168,1000,64.856566
+5540.281263,1000,65.908327
+5565.711141,1000,66.962104
+5556.512835,1000,68.013427
+5569.806855,1000,69.060416
+5563.633497,1000,70.071406
+5567.37055,1000,71.073722
+5569.296498,1000,72.092269
+5585.95279,1000,73.119905
+5562.970955,1000,74.162285
+5567.092114,1000,75.203389
+5542.583236,1000,76.255565
+5584.622661,1000,77.293009
+5589.693622,1000,78.333484
+5570.931373,1000,79.374914
+5577.068572,1000,80.417229
+5544.344188,1000,81.462481
+5588.523099,1000,82.507497
+5571.743971,1000,83.555671
+5563.25962,1000,84.599184
+5572.492678,1000,85.640658
+5604.837281,1000,86.677407
+5579.576886,1000,87.718099
+5542.78444,1000,88.763794
+5577.64076,1000,89.802948
+5534.055427,1000,90.848242
+5566.608979,1000,91.890243
+5548.491408,1000,92.93387
+5595.331925,1000,93.975235
+5567.715658,1000,95.019204
+5576.46651,1000,96.061959
+5576.761717,1000,97.102602
+5571.387384,1000,98.147914
+5559.685698,1000,99.194083
+5577.240639,1000,100.236856
+5572.679925,1000,101.27986
+5572.047691,1000,102.327862
+5569.06556,1000,103.371852
+5544.143835,1000,104.419713
+5574.264787,1000,105.466611
+5594.264005,1000,106.509205
+5575.707679,1000,107.553529
+5568.716602,1000,108.606632
+5588.560825,1000,109.649822
+5560.117436,1000,110.701463
+5549.528032,1000,111.752786
+5551.613105,1000,112.800858
+5556.93718,1000,113.849758
+5583.017121,1000,114.893378
+5550.923894,1000,115.940132
+5583.925752,1000,116.98191
+5563.217176,1000,118.027358
+5568.783899,1000,119.066672
+5564.37095,1000,120.110978
+5586.429606,1000,121.157058
+5572.377408,1000,122.200592
+5563.953028,1000,123.245982
+5570.548952,1000,124.290761
+5569.991771,1000,125.334872
+5549.923183,1000,126.384363
+5567.786066,1000,127.427726
+5542.909656,1000,128.483996
+5580.588716,1000,129.52423
+5564.617737,1000,130.557464
+5562.132102,1000,131.563036
+5530.838298,1000,132.583936
+5570.773813,1000,133.604815
+5572.958868,1000,134.656435
+5573.465718,1000,135.705783
+5562.720358,1000,136.754902
+5572.76866,1000,137.802345
+5575.070776,1000,138.844315
+5557.684079,1000,139.886634
+5542.396176,1000,140.935772
+5574.08599,1000,141.98236
+5576.720636,1000,143.032866
+5566.067424,1000,144.076231
+5571.086831,1000,145.119242
+5551.604918,1000,146.168159
+5537.28939,1000,147.221932
+5569.537861,1000,148.266629
+5564.126632,1000,149.309789
+5562.545132,1000,150.352092
+5568.346308,1000,151.392841
+5589.201428,1000,152.42522
+5551.735864,1000,153.467628
+5567.478402,1000,154.514002
diff --git a/logs/benchmark/td3-Swimmer-v3/0.monitor.csv b/logs/benchmark/td3-Swimmer-v3/0.monitor.csv
new file mode 100644
index 000000000..3038a9eb3
--- /dev/null
+++ b/logs/benchmark/td3-Swimmer-v3/0.monitor.csv
@@ -0,0 +1,152 @@
+#{"t_start": 1637000774.2236927, "env_id": "Swimmer-v3"}
+r,l,t
+359.641789,1000,2.495278
+359.570296,1000,3.071362
+360.113881,1000,3.6409
+359.03474,1000,4.213458
+357.394025,1000,4.784713
+358.59928,1000,5.358865
+357.430308,1000,5.931267
+358.804166,1000,6.50682
+359.546781,1000,7.078298
+360.369316,1000,7.651718
+358.814501,1000,8.225361
+360.523946,1000,8.800662
+358.808276,1000,9.371002
+358.636895,1000,9.945042
+357.534344,1000,10.519945
+357.954519,1000,11.091039
+359.894878,1000,11.662415
+358.801778,1000,12.234455
+358.480254,1000,12.806212
+359.203047,1000,13.380753
+359.876706,1000,13.952598
+359.312381,1000,14.524766
+359.680299,1000,15.096222
+359.043945,1000,15.671176
+359.679795,1000,16.241651
+359.483661,1000,16.814119
+359.758249,1000,17.385462
+359.811952,1000,17.95867
+358.561262,1000,18.5307
+357.412488,1000,19.104124
+360.872511,1000,19.67549
+357.815131,1000,20.249243
+360.131936,1000,20.819992
+359.50243,1000,21.39925
+357.950284,1000,21.972713
+358.424513,1000,22.547823
+359.138897,1000,23.118529
+357.694483,1000,23.69503
+360.333013,1000,24.267218
+359.056815,1000,24.842595
+357.677642,1000,25.41583
+362.185269,1000,25.99567
+357.90984,1000,26.567426
+358.907786,1000,27.143991
+360.265217,1000,27.715817
+358.350687,1000,28.292432
+359.73469,1000,28.862813
+359.777528,1000,29.441296
+358.585794,1000,30.009815
+356.470225,1000,30.590181
+359.021703,1000,31.161805
+359.265097,1000,31.739443
+362.155727,1000,32.311267
+359.150616,1000,32.889125
+360.313185,1000,33.458539
+360.148184,1000,34.033376
+357.603206,1000,34.602303
+359.633595,1000,35.183229
+357.800765,1000,35.752847
+359.660842,1000,36.325576
+356.895054,1000,36.896641
+356.561071,1000,37.472333
+359.436246,1000,38.041705
+363.682589,1000,38.619516
+361.20847,1000,39.188705
+360.192182,1000,39.762878
+358.750261,1000,40.333675
+359.691724,1000,40.90725
+359.746504,1000,41.477144
+359.672244,1000,42.054115
+360.707628,1000,42.625475
+360.007404,1000,43.201945
+358.718568,1000,43.774617
+357.882352,1000,44.349422
+357.641974,1000,44.918798
+357.828032,1000,45.496988
+360.783593,1000,46.072324
+358.932797,1000,46.651056
+359.322452,1000,47.225838
+359.962691,1000,47.805912
+357.9932,1000,48.382297
+358.256472,1000,48.962675
+360.118705,1000,49.542459
+358.425265,1000,50.134149
+358.770482,1000,50.711005
+358.481121,1000,51.300118
+359.533601,1000,51.872824
+355.129853,1000,52.458704
+358.804439,1000,53.035775
+362.672419,1000,53.614137
+360.426408,1000,54.190934
+358.218543,1000,54.771804
+359.20479,1000,55.350816
+358.046392,1000,55.930684
+360.124082,1000,56.508608
+357.347052,1000,57.091463
+358.670004,1000,57.667814
+358.747429,1000,58.248147
+357.862899,1000,58.826074
+357.880015,1000,59.407699
+359.927678,1000,59.983092
+360.5678,1000,60.561925
+359.100832,1000,61.139072
+358.005559,1000,61.716914
+357.345316,1000,62.292263
+360.320859,1000,62.870202
+360.106455,1000,63.44449
+359.312299,1000,64.022288
+358.367861,1000,64.59663
+358.233116,1000,65.177312
+360.114876,1000,65.750304
+362.564674,1000,66.328007
+357.748302,1000,66.903903
+358.134657,1000,67.482774
+359.10814,1000,68.056764
+360.724803,1000,68.636815
+360.233386,1000,69.21632
+359.379633,1000,69.795586
+359.109237,1000,70.374286
+360.187096,1000,70.951621
+361.007734,1000,71.531312
+357.200745,1000,72.111383
+357.802925,1000,72.691016
+357.958293,1000,73.270766
+358.752565,1000,73.849949
+357.853371,1000,74.429569
+359.933883,1000,75.007288
+358.500272,1000,75.586137
+359.925086,1000,76.165465
+359.984568,1000,76.746386
+359.852355,1000,77.325752
+358.19038,1000,77.906304
+359.548422,1000,78.483931
+360.175419,1000,79.062699
+357.867164,1000,79.642133
+358.996143,1000,80.220293
+359.125398,1000,80.800538
+358.282459,1000,81.380596
+360.140719,1000,81.959782
+359.55373,1000,82.538762
+359.372449,1000,83.119266
+358.863364,1000,83.698115
+359.512481,1000,84.277126
+360.722991,1000,84.861782
+357.808485,1000,85.442258
+360.823578,1000,86.023095
+358.531012,1000,86.602623
+357.589105,1000,87.183789
+358.238164,1000,87.765451
+358.700401,1000,88.357664
diff --git a/logs/benchmark/td3-Walker2d-v3/0.monitor.csv b/logs/benchmark/td3-Walker2d-v3/0.monitor.csv
new file mode 100644
index 000000000..e4daba9d3
--- /dev/null
+++ b/logs/benchmark/td3-Walker2d-v3/0.monitor.csv
@@ -0,0 +1,152 @@
+#{"t_start": 1637079111.5854242, "env_id": "Walker2d-v3"}
+r,l,t
+4713.222342,1000,2.925961
+4699.898517,1000,3.418506
+4751.937629,1000,3.911664
+4706.483733,1000,4.420207
+4655.744895,1000,4.929009
+4746.518592,1000,5.422398
+4653.552667,1000,5.916017
+4672.84818,1000,6.408116
+4703.476567,1000,6.901835
+4727.045364,1000,7.396931
+4764.48425,1000,7.899874
+4673.934933,1000,8.402458
+4735.89275,1000,8.905669
+4681.448293,1000,9.404801
+4663.419012,1000,9.906479
+4741.036068,1000,10.423858
+4776.998616,1000,10.918464
+4807.280352,1000,11.410323
+4715.0159,1000,11.904002
+4822.546004,1000,12.395902
+4671.662022,1000,12.889747
+4652.336249,1000,13.387347
+4695.533832,1000,13.892613
+4847.166833,1000,14.394251
+4655.185549,1000,14.895613
+4688.498451,1000,15.389625
+4739.705215,1000,15.882482
+4666.770331,1000,16.374839
+4693.67737,1000,16.868248
+4764.955489,1000,17.36008
+4686.974181,1000,17.853378
+4820.182533,1000,18.345613
+4845.166842,1000,18.838337
+4787.443131,1000,19.330749
+4690.483163,1000,19.823926
+4665.312593,1000,20.315876
+4679.686161,1000,20.809337
+4700.821125,1000,21.303317
+4782.670707,1000,21.796154
+4706.786564,1000,22.28817
+4786.98368,1000,22.781069
+4755.211489,1000,23.272796
+4677.057597,1000,23.766083
+4704.889731,1000,24.260198
+4728.140208,1000,24.75349
+4700.355258,1000,25.24575
+4647.354657,1000,25.739137
+4655.331001,1000,26.231366
+4726.369874,1000,26.724899
+4741.387625,1000,27.219255
+4678.277027,1000,27.712617
+4677.802177,1000,28.205392
+4681.481501,1000,28.698767
+4702.585323,1000,29.190739
+4759.32336,1000,29.698264
+4770.687229,1000,30.196281
+4695.893025,1000,30.691707
+4676.861166,1000,31.184657
+4670.50292,1000,31.677967
+4764.270237,1000,32.169342
+4787.185245,1000,32.662223
+4668.448923,1000,33.154175
+4713.968403,1000,33.654163
+4718.385203,1000,34.14939
+4697.510377,1000,34.648995
+4683.077978,1000,35.150262
+4747.644041,1000,35.654105
+4747.19109,1000,36.158213
+4806.405801,1000,36.663706
+4723.541921,1000,37.163387
+4764.209517,1000,37.669141
+4724.897734,1000,38.168467
+4679.092953,1000,38.67325
+4679.095129,1000,39.180405
+4764.894746,1000,39.684908
+4696.914022,1000,40.179095
+4699.384799,1000,40.67243
+4665.509667,1000,41.164933
+4658.561783,1000,41.659124
+4732.475967,1000,42.151364
+4732.677069,1000,42.645196
+4725.009965,1000,43.137586
+4771.145801,1000,43.630772
+4789.922395,1000,44.122984
+4741.260781,1000,44.616999
+4779.882372,1000,45.109736
+4669.111133,1000,45.60335
+4735.982985,1000,46.095567
+4656.794656,1000,46.589236
+4705.046318,1000,47.081486
+4667.169479,1000,47.57493
+4727.570676,1000,48.066895
+4753.8078,1000,48.560139
+4723.444191,1000,49.05258
+4650.243397,1000,49.546246
+4749.080268,1000,50.038034
+4682.820309,1000,50.531625
+4750.23193,1000,51.02326
+4681.531485,1000,51.516833
+4732.756266,1000,52.008637
+4735.991289,1000,52.502037
+4711.357831,1000,52.994057
+4692.290058,1000,53.487376
+4697.980463,1000,53.979545
+4712.576324,1000,54.47312
+4707.939111,1000,54.965546
+4676.805636,1000,55.458857
+4693.678467,1000,55.950809
+4652.259965,1000,56.444952
+4727.580116,1000,56.936576
+4687.330104,1000,57.429811
+4720.00266,1000,57.921388
+4727.579143,1000,58.414617
+4840.602934,1000,58.90617
+4699.903584,1000,59.399471
+4683.63935,1000,59.891608
+4817.064508,1000,60.384578
+4703.445304,1000,60.876375
+4713.265769,1000,61.369777
+4656.727961,1000,61.865175
+4728.888129,1000,62.361873
+4797.909774,1000,62.877674
+4713.129963,1000,63.393237
+4757.899931,1000,63.919626
+4694.440488,1000,64.449241
+4689.175879,1000,64.96671
+4717.002461,1000,65.51205
+4671.210642,1000,66.122687
+4677.595751,1000,66.750378
+4684.863419,1000,67.37855
+4767.789955,1000,68.007272
+4680.607365,1000,68.635993
+4734.955648,1000,69.266129
+4733.692232,1000,69.893781
+4739.518362,1000,70.521743
+4710.129456,1000,71.151237
+4725.400459,1000,71.779165
+4689.418861,1000,72.314024
+4769.802058,1000,72.80852
+4661.856565,1000,73.300351
+4678.706479,1000,73.793431
+4665.207694,1000,74.285826
+4708.714693,1000,74.779163
+4713.938072,1000,75.271038
+4632.684674,1000,75.76447
+4770.354922,1000,76.255954
+4797.613838,1000,76.748758
+4674.255724,1000,77.240762
+4816.314847,1000,77.733967
+4773.944102,1000,78.225535
diff --git a/logs/benchmark/tqc-Ant-v3/0.monitor.csv b/logs/benchmark/tqc-Ant-v3/0.monitor.csv
new file mode 100644
index 000000000..a5679b7c0
--- /dev/null
+++ b/logs/benchmark/tqc-Ant-v3/0.monitor.csv
@@ -0,0 +1,204 @@
+#{"t_start": 1635520025.1870697, "env_id": "Ant-v3"}
+r,l,t
+5534.7148,1000,1.23041
+4756.65456,1000,2.616743
+5370.455958,1000,4.485943
+5127.793559,1000,6.006166
+-679.548542,1000,7.513026
+887.417787,1000,9.57612
+5236.610944,1000,11.117907
+5114.330424,1000,12.293732
+3115.002809,1000,13.505234
+-294.649237,1000,14.930072
+5172.293703,1000,16.295625
+5100.155208,1000,17.486366
+1589.210775,326,17.884528
+-447.800689,1000,19.599654
+5380.178601,1000,20.836229
+8.661818,11,20.849121
+5190.376365,1000,22.036417
+761.249665,1000,23.546703
+5172.433279,1000,25.483287
+25.092454,18,25.531194
+1022.979551,1000,27.605027
+5139.694602,1000,29.585836
+4651.170587,918,32.193597
+3851.893128,1000,34.004291
+5073.170282,1000,35.349819
+1463.956655,294,35.742164
+5309.67167,1000,37.026778
+3808.426195,755,38.070721
+2492.476411,460,38.7715
+815.848795,171,39.021666
+461.189731,112,39.190935
+5225.980614,1000,40.768495
+4970.804821,1000,42.243897
+5106.933019,1000,43.73883
+3527.436174,729,44.852793
+5127.912137,1000,46.371786
+2153.330152,1000,47.892141
+2488.410102,1000,49.23941
+4955.697323,1000,50.54546
+1831.519011,1000,51.847296
+280.58114,66,51.921345
+5299.161342,1000,53.182419
+448.584118,1000,54.525936
+5159.988086,985,55.718893
+41.744567,26,55.749979
+4970.804047,1000,57.009198
+741.836332,179,57.229602
+2776.726865,614,57.987356
+5354.052178,1000,59.267164
+1753.759864,354,59.674475
+5075.281119,1000,60.835646
+5148.239122,1000,62.147441
+5005.598175,1000,63.923539
+5127.769923,1000,65.649105
+2812.850828,602,66.378118
+5019.210263,1000,67.529924
+2046.134898,1000,68.694861
+5173.813385,1000,69.816093
+5044.347351,1000,70.926511
+1943.412649,352,71.326432
+5108.210674,1000,72.464347
+5271.751501,1000,73.580101
+855.828288,1000,74.735223
+2167.631462,423,75.204097
+4849.346662,1000,76.356214
+5113.383598,1000,77.470554
+5330.82087,1000,78.598749
+454.846377,97,78.703593
+1061.998627,213,78.943831
+5024.823794,1000,80.057956
+5266.609914,1000,81.169443
+1093.532092,217,81.412035
+2729.080246,537,82.024955
+1538.316466,294,82.356521
+2585.795407,519,82.939113
+5023.994115,1000,84.042375
+4762.671813,935,85.088158
+709.478638,1000,86.254657
+4183.587729,791,87.131607
+5119.598561,1000,88.23278
+3880.863842,1000,89.374039
+5020.706058,1000,90.486637
+5317.89856,1000,91.588877
+1376.126217,280,91.907407
+5200.797136,1000,93.025255
+5312.061625,1000,94.138285
+1494.084865,295,94.459565
+5134.765655,1000,95.579214
+3182.790614,1000,96.745546
+5308.633541,1000,97.869965
+1055.129837,209,98.118725
+3588.663329,723,98.945638
+3436.846361,676,99.699689
+5316.567153,1000,100.816853
+3220.702373,1000,101.948633
+5317.171544,1000,103.072016
+754.222491,191,103.280725
+5105.190434,1000,104.410252
+2357.762748,462,104.93487
+5263.616454,1000,106.070261
+1627.623093,304,106.418392
+1197.640295,246,106.703165
+3809.554705,1000,107.859984
+4878.238482,948,108.913471
+2466.201872,463,109.511661
+167.557303,53,109.59631
+719.340805,156,109.840223
+4433.321222,836,110.911351
+5259.335539,1000,112.032591
+5386.82686,1000,113.155468
+5227.514111,1000,114.255355
+1214.856521,249,114.537736
+543.513195,111,114.662953
+4.52582,11,114.675663
+1844.219597,377,115.093257
+439.948527,107,115.215193
+5278.266144,1000,116.322196
+5030.511722,1000,117.433135
+231.233256,73,117.518499
+1574.36236,321,117.921484
+1765.473827,363,118.363215
+2610.94502,566,119.006536
+2392.877739,440,119.49802
+5036.845093,1000,120.616833
+4920.351945,933,121.653499
+5242.160915,1000,122.766917
+5258.964084,1000,123.875498
+72.91576,35,123.913788
+5032.908664,1000,125.014812
+5493.784828,1000,126.133094
+4339.560139,786,127.002325
+1334.546196,289,127.322829
+711.149816,147,127.488555
+5127.289145,1000,128.605871
+2567.844695,1000,129.783035
+5204.136177,1000,130.892755
+5189.857011,1000,132.001152
+2013.439507,1000,133.195715
+5343.789675,1000,134.313677
+5336.590434,1000,135.416339
+4975.332482,1000,136.531208
+1555.475678,297,136.866615
+1158.318805,256,137.146907
+48.938617,39,137.191485
+80.477729,37,137.234152
+5202.153171,1000,138.348277
+3774.503833,1000,139.496088
+4140.106521,799,140.39589
+5284.665049,990,141.494034
+5194.186719,1000,142.622844
+5128.867121,1000,143.738394
+1227.371802,246,144.008487
+1421.616943,299,144.341559
+2240.899497,434,144.821916
+5150.876733,1000,145.92746
+1325.549371,1000,147.159573
+618.801866,165,147.343054
+492.216439,1000,148.486616
+5079.98781,1000,149.610084
+5207.624798,1000,150.714473
+5138.205593,1000,151.829123
+4572.933278,1000,152.950248
+4986.891165,1000,154.068689
+5218.100424,1000,155.179108
+4363.077675,1000,156.310597
+1219.915169,1000,157.516976
+1798.343513,374,157.946175
+4371.543869,1000,159.07089
+5364.523654,1000,160.176045
+5105.42255,1000,161.292292
+5082.459433,1000,162.420067
+5124.211873,1000,163.538993
+620.665607,145,163.703256
+393.233947,95,163.811109
+4990.775851,1000,164.952743
+10.208523,12,164.966665
+5193.34074,1000,166.091385
+5225.167657,1000,167.223278
+1571.64985,311,167.569145
+729.890263,166,167.756843
+5283.945115,1000,168.860109
+4930.009787,1000,169.975033
+5185.489689,1000,171.080436
+1363.714909,279,171.392022
+2491.71162,485,171.926673
+5394.261012,1000,173.033077
+1591.310311,327,173.396305
+2309.898354,1000,174.581627
+2023.758642,396,175.025765
+5441.617348,1000,176.136416
+5389.003992,1000,177.25053
+3628.210976,692,178.032979
+5186.023368,1000,179.142866
+1244.244991,1000,180.321612
+1725.307461,340,180.697076
+1872.056791,381,181.122084
+86.938135,44,181.172227
+4987.412333,954,182.228489
+1791.806893,362,182.631286
+5113.776183,1000,183.738716
+5066.432996,1000,184.842623
+6.76629,10,184.854404
diff --git a/logs/benchmark/tqc-HalfCheetah-v3/0.monitor.csv b/logs/benchmark/tqc-HalfCheetah-v3/0.monitor.csv
new file mode 100644
index 000000000..1cafc35a3
--- /dev/null
+++ b/logs/benchmark/tqc-HalfCheetah-v3/0.monitor.csv
@@ -0,0 +1,152 @@
+#{"t_start": 1635520314.1219075, "env_id": "HalfCheetah-v3"}
+r,l,t
+12087.545739,1000,0.635621
+12048.857315,1000,1.256774
+12139.620892,1000,1.886295
+11993.557437,1000,2.518023
+11770.338384,1000,3.149676
+11833.168177,1000,3.778083
+12212.825461,1000,4.413147
+12281.952721,1000,5.041774
+12046.963418,1000,5.678325
+12206.406565,1000,6.313848
+12082.597242,1000,6.944579
+12101.909152,1000,7.576967
+11991.7988,1000,8.201665
+12098.798176,1000,8.83519
+12111.020112,1000,9.467347
+12007.589765,1000,10.096464
+12003.200135,1000,10.726595
+12254.779729,1000,11.350649
+12056.815216,1000,12.00578
+12057.843972,1000,12.649585
+11950.228529,1000,13.273728
+12278.347653,1000,13.905694
+12121.69685,1000,14.53669
+12105.216894,1000,15.167826
+12054.66286,1000,15.787573
+11966.326799,1000,16.422558
+12020.662188,1000,17.060878
+12200.931437,1000,17.682687
+12139.881535,1000,18.315055
+12061.278718,1000,18.935845
+11977.043728,1000,19.565012
+12180.371677,1000,20.203279
+11921.687765,1000,20.846633
+12106.134416,1000,21.486119
+12130.825291,1000,22.114014
+12006.842946,1000,22.75813
+12123.790577,1000,23.499815
+12245.468803,1000,24.212706
+11833.784738,1000,24.902409
+12227.778749,1000,25.647418
+12145.420723,1000,26.326969
+12125.567702,1000,26.988612
+11985.623023,1000,27.775185
+12151.860401,1000,28.471773
+12127.786908,1000,29.204709
+12219.992303,1000,29.961596
+12023.756213,1000,30.76746
+12033.988887,1000,31.542756
+12278.462364,1000,32.243964
+11869.89925,1000,32.934071
+12103.279955,1000,33.722426
+11980.274466,1000,34.538253
+12229.5988,1000,35.345716
+12087.820098,1000,36.194876
+12055.1683,1000,37.006814
+12006.211276,1000,37.804957
+12033.50515,1000,38.660148
+11883.265278,1000,39.336931
+12209.850208,1000,40.072577
+12083.855462,1000,40.80578
+11982.904542,1000,41.566595
+12208.764265,1000,42.287383
+12087.942607,1000,42.968348
+11956.480023,1000,43.651125
+12010.503145,1000,44.365071
+12251.81143,1000,45.114635
+12180.919549,1000,45.789142
+11924.106653,1000,46.490635
+12376.272259,1000,47.234792
+12392.176575,1000,47.920702
+12160.560183,1000,48.692303
+12120.98901,1000,49.653049
+12164.904071,1000,50.511473
+12129.831703,1000,51.185885
+11851.762261,1000,51.836827
+11865.609902,1000,52.645839
+12190.61455,1000,53.366449
+11943.746276,1000,54.063504
+11989.094221,1000,54.731442
+12276.354568,1000,55.469948
+12152.670297,1000,56.527557
+12246.675321,1000,57.288417
+12059.976467,1000,58.068702
+12054.202772,1000,58.81217
+12103.991513,1000,59.477695
+12018.710454,1000,60.186614
+11869.525921,1000,60.916745
+11840.399973,1000,61.63063
+12135.26023,1000,62.35434
+12235.362361,1000,63.000652
+12090.75412,1000,63.670268
+12199.604864,1000,64.32541
+11765.712813,1000,64.972616
+11959.055767,1000,65.620297
+12063.411563,1000,66.269301
+11936.52307,1000,66.971538
+11929.602238,1000,67.641373
+12268.43669,1000,68.309781
+12160.072066,1000,68.984849
+12337.729959,1000,69.705093
+12086.161691,1000,70.434092
+12182.826781,1000,71.091367
+12110.208585,1000,71.741229
+12174.112825,1000,72.428269
+12229.192183,1000,73.163661
+12069.521798,1000,73.941696
+12167.739852,1000,74.614265
+12199.887649,1000,75.264695
+12224.117273,1000,76.235806
+12174.623678,1000,77.013226
+12165.585073,1000,77.723206
+12130.625928,1000,78.423515
+12195.124211,1000,79.102236
+12007.311613,1000,79.785497
+12120.201799,1000,80.756801
+12111.111229,1000,81.628576
+12196.135553,1000,82.332743
+12047.679173,1000,82.978592
+11816.10298,1000,83.628387
+12008.279972,1000,84.301305
+12213.405324,1000,84.982671
+12183.858166,1000,85.897329
+12182.403425,1000,86.90418
+12243.308872,1000,87.932353
+12165.492766,1000,88.798385
+11956.89639,1000,89.533665
+11994.356447,1000,90.294686
+12082.574326,1000,90.996723
+11713.792191,1000,91.655463
+12021.301947,1000,92.307029
+12294.437639,1000,92.953017
+11922.566854,1000,93.603896
+11874.722555,1000,94.426775
+12030.689651,1000,95.197312
+12084.705636,1000,95.966095
+12055.270149,1000,96.628884
+12261.111535,1000,97.28001
+12135.344728,1000,98.060944
+12180.922069,1000,98.749754
+12064.36318,1000,99.439328
+12235.741592,1000,100.15353
+11977.196173,1000,100.823949
+12219.579365,1000,101.499903
+12199.28199,1000,102.185532
+12215.748006,1000,102.868353
+11986.682321,1000,103.537081
+11985.689879,1000,104.40693
+12091.843181,1000,105.413648
+12151.821998,1000,106.398839
+12180.453859,1000,107.499574
diff --git a/logs/benchmark/tqc-Hopper-v3/0.monitor.csv b/logs/benchmark/tqc-Hopper-v3/0.monitor.csv
new file mode 100644
index 000000000..dcff66143
--- /dev/null
+++ b/logs/benchmark/tqc-Hopper-v3/0.monitor.csv
@@ -0,0 +1,152 @@
+#{"t_start": 1635520596.9407363, "env_id": "Hopper-v3"}
+r,l,t
+3747.258698,1000,0.738512
+3755.334143,1000,1.513512
+3753.670311,1000,2.291987
+3749.724932,1000,3.056552
+3754.938174,1000,3.858865
+3760.170622,1000,4.641201
+3743.994437,1000,5.444073
+3751.034857,1000,6.311399
+3761.024959,1000,7.179154
+3742.662985,1000,7.987041
+3763.354561,1000,8.796961
+3755.485831,1000,9.61046
+3752.689013,1000,10.441958
+3757.806506,1000,11.233017
+3739.630292,1000,11.97938
+3771.787771,1000,12.737068
+3758.995596,1000,13.50461
+3763.334611,1000,14.2626
+3756.835718,1000,15.053847
+3755.088796,1000,15.817265
+3753.076756,1000,16.574713
+3755.510477,1000,17.337818
+3745.806373,1000,18.122683
+3751.575562,1000,18.937137
+3768.367396,1000,19.703224
+3762.994898,1000,20.459802
+3757.171834,1000,21.233851
+3762.498209,1000,22.009166
+3722.311836,1000,22.801257
+3754.501534,1000,23.608131
+3748.64779,1000,24.390429
+3772.552868,1000,25.177464
+3751.927189,1000,25.953676
+3751.369549,1000,26.748655
+3752.597571,1000,27.536605
+3762.000193,1000,28.326548
+3750.030194,1000,29.128336
+3751.632263,1000,29.917194
+3748.003185,1000,30.713974
+3742.470417,1000,31.492959
+3761.636008,1000,32.271177
+3771.826146,1000,33.087138
+3760.522164,1000,33.879877
+3749.410629,1000,34.658016
+3753.123434,1000,35.438123
+3747.074272,1000,36.207246
+3747.639276,1000,37.019146
+3746.795069,1000,37.817367
+3740.818708,1000,38.61681
+3749.355975,1000,39.418067
+3759.324213,1000,40.197731
+3752.35616,1000,40.999974
+3759.649575,1000,41.759511
+3753.880148,1000,42.534013
+3737.993013,1000,43.348872
+3738.760659,1000,44.153023
+3754.603499,1000,44.943743
+3754.461636,1000,45.712808
+3749.566784,1000,46.49998
+3744.452677,1000,47.290966
+3736.226596,1000,48.118804
+3755.508181,1000,48.905542
+3770.105935,1000,49.698911
+3763.569103,1000,50.489965
+3751.900951,1000,51.340637
+3756.708346,1000,52.172738
+3736.356173,1000,52.990861
+3758.110715,1000,53.762442
+3765.47644,1000,54.52216
+3753.38158,1000,55.644922
+3761.252167,1000,56.843671
+3761.83225,1000,57.677159
+3753.434224,1000,58.467593
+3742.517071,1000,59.363004
+3751.72393,1000,60.266062
+3745.593315,1000,61.893217
+3765.038157,1000,63.062915
+3751.552338,1000,64.157699
+3759.480461,1000,65.559391
+3748.721507,1000,66.538214
+3759.724273,1000,67.634554
+3757.872072,1000,69.189422
+3747.972209,1000,70.402823
+3761.910449,1000,72.026053
+3752.239387,1000,73.123342
+3762.794416,1000,73.994401
+3755.384681,1000,74.789994
+3747.080512,1000,75.563806
+3754.133303,1000,76.315958
+3752.204949,1000,77.157524
+3758.760632,1000,77.974916
+3762.256087,1000,78.749856
+3754.718614,1000,79.564297
+3757.285378,1000,80.509527
+3750.229627,1000,81.457074
+3765.680314,1000,82.262141
+3749.246509,1000,83.02722
+3741.745151,1000,83.896468
+3758.472173,1000,84.697689
+3743.74413,1000,85.491299
+3744.417895,1000,86.283593
+3750.589064,1000,87.154768
+3758.722476,1000,88.015409
+3771.698478,1000,88.995306
+3755.300133,1000,89.855249
+3746.793599,1000,91.16388
+3759.06877,1000,92.318864
+3749.18274,1000,93.566354
+3752.473648,1000,94.695982
+3765.583352,1000,95.723222
+3754.855191,1000,96.964406
+3761.908745,1000,97.94882
+3761.81778,1000,98.818065
+3758.652935,1000,99.912592
+3764.710834,1000,100.9313
+3757.623832,1000,101.759024
+3760.540148,1000,102.695414
+3747.396653,1000,103.499059
+3755.178719,1000,104.300949
+3742.448797,1000,105.122467
+3760.653955,1000,105.94513
+3746.102567,1000,106.833105
+3768.552229,1000,107.658326
+3758.123986,1000,108.446702
+3748.508376,1000,109.29838
+3740.015206,1000,110.120737
+3757.874748,1000,111.045687
+3766.486781,1000,111.922093
+3757.843996,1000,112.742225
+3749.071908,1000,113.519075
+3742.971825,1000,114.316193
+3759.60032,1000,115.112966
+3765.435866,1000,115.936625
+3746.459052,1000,116.726472
+3749.586958,1000,117.516387
+3768.231335,1000,118.282126
+3756.737566,1000,119.052252
+3756.197588,1000,119.810227
+3751.177301,1000,120.601123
+3747.749014,1000,121.388769
+3760.834439,1000,122.18234
+3751.834988,1000,122.985348
+3765.738381,1000,123.751654
+3754.510388,1000,124.521034
+3750.644235,1000,125.309533
+3752.75092,1000,126.107251
+3761.535395,1000,126.915647
+3752.359245,1000,127.703813
+3736.880008,1000,128.47537
+3751.364432,1000,129.286715
diff --git a/logs/benchmark/tqc-Humanoid-v3/0.monitor.csv b/logs/benchmark/tqc-Humanoid-v3/0.monitor.csv
new file mode 100644
index 000000000..cbd8936d9
--- /dev/null
+++ b/logs/benchmark/tqc-Humanoid-v3/0.monitor.csv
@@ -0,0 +1,167 @@
+#{"t_start": 1636991995.409108, "env_id": "Humanoid-v3"}
+r,l,t
+8082.959108,1000,3.318235
+8026.578582,1000,4.722991
+8113.713379,1000,6.122962
+3498.763138,456,6.761889
+2350.097505,312,7.199975
+6307.512273,798,8.320259
+8018.879588,1000,9.708417
+8075.568955,1000,11.100669
+8104.137413,1000,12.238547
+8081.779097,1000,13.375025
+4682.215431,619,14.118629
+8095.709506,1000,15.512217
+8101.056083,1000,16.897889
+8109.065824,1000,18.275461
+7912.649101,984,19.63352
+8038.032483,1000,21.009068
+8041.582963,1000,22.382195
+7888.783078,1000,23.757031
+8010.874623,1000,25.130846
+8022.802392,1000,26.498757
+8067.195472,1000,27.873432
+1635.939313,227,28.184064
+1887.452423,261,28.540952
+8074.893119,1000,29.918613
+8082.129068,1000,31.291587
+4611.893804,594,32.108148
+7988.732755,1000,33.479492
+8087.013595,1000,34.855587
+2234.8902,302,35.270726
+4567.160018,582,36.07229
+8037.779014,1000,37.448334
+8090.102467,1000,38.827615
+8068.408689,1000,40.20276
+5517.865944,704,41.169808
+7958.254926,1000,42.539181
+4306.094621,548,43.295134
+8050.925357,1000,44.666637
+8063.185807,1000,46.044038
+8114.752537,1000,47.418578
+4592.851068,602,48.244835
+8103.222928,1000,49.617112
+8096.377868,1000,50.990821
+8070.998553,1000,52.359002
+7858.146587,1000,53.732527
+8075.3459,1000,55.094871
+8077.700449,1000,56.47593
+8061.289735,1000,57.85194
+8088.360188,1000,59.22872
+8084.32056,1000,60.606941
+8098.370952,1000,61.986589
+8075.51546,1000,63.371551
+8060.794835,1000,64.747331
+7932.321643,1000,66.124092
+4552.695599,605,66.956859
+8041.635399,1000,68.333705
+2290.215837,307,68.757674
+8030.867598,1000,70.129375
+5474.875133,684,71.071555
+7983.180535,1000,72.448982
+4718.784347,610,73.291767
+7707.111868,1000,74.53432
+7917.674938,1000,75.656321
+8083.300948,1000,76.792105
+7644.208776,963,78.119369
+8026.91681,1000,79.508715
+8051.303627,1000,80.890348
+8079.954842,1000,82.266363
+8045.349661,1000,83.636113
+3916.431296,502,84.33098
+8099.432636,1000,85.710615
+7755.207636,1000,87.088174
+8065.267939,1000,88.467613
+8051.591498,1000,89.848705
+8100.071944,1000,91.229072
+8045.933226,1000,92.607033
+3721.480954,485,93.275266
+7941.877401,1000,94.652558
+8079.204851,1000,96.029969
+8022.663672,1000,97.41042
+8054.058494,1000,98.788603
+8104.10514,1000,100.166517
+8058.498378,1000,101.544111
+7936.112477,1000,102.921241
+5337.9408,690,103.870347
+8077.60258,1000,105.243143
+5981.932322,771,106.305215
+7894.989983,1000,107.676043
+8021.809825,1000,109.054561
+8066.574782,1000,110.433227
+7922.102002,1000,111.804561
+8057.166473,1000,113.183712
+8003.564645,1000,114.553694
+8047.120984,1000,115.930435
+7893.441595,1000,117.301714
+6508.107121,814,118.42377
+8034.047566,1000,119.799596
+8076.899488,1000,121.17196
+7935.186264,1000,122.539256
+8042.57763,1000,123.912429
+7895.568931,1000,125.286333
+8055.578719,1000,126.65864
+8021.812615,1000,128.034223
+8067.678856,1000,129.409215
+7683.230877,952,130.719733
+8041.771777,1000,132.090345
+8009.753257,1000,133.461366
+8094.302434,1000,134.838326
+8046.503797,1000,136.209374
+5630.973149,722,137.155318
+8079.25966,1000,138.298433
+8000.315864,1000,139.435905
+8001.210462,1000,140.747525
+7627.514818,1000,142.117249
+8085.164463,1000,143.489918
+7918.620981,1000,144.868665
+5785.528185,728,145.871318
+2323.732921,307,146.29511
+8087.301534,1000,147.672897
+7837.867489,1000,149.042773
+8101.167401,1000,150.421709
+7909.278696,1000,151.795975
+8003.824195,1000,153.168923
+8069.445316,1000,154.540384
+3811.291725,488,155.206601
+7972.141205,1000,156.580011
+4140.644777,529,157.304013
+7992.172428,1000,158.680135
+3973.766262,506,159.385895
+2333.178481,316,159.828336
+8067.674404,1000,161.2173
+8096.763693,1000,162.607967
+8075.401528,1000,163.995308
+7002.100637,872,165.205219
+8059.977257,1000,166.589516
+8038.291812,1000,167.977695
+8070.457823,1000,169.363459
+8044.431504,1000,170.746339
+7965.238817,1000,172.134696
+8056.431887,1000,173.524283
+7997.63439,1000,174.901771
+8070.85515,1000,176.282644
+8069.846653,1000,177.672074
+7914.451179,1000,179.062662
+7909.572762,1000,180.452874
+2204.1478,299,180.871854
+8058.632121,1000,182.263204
+7663.225857,956,183.588003
+4088.693157,523,184.322219
+7899.644842,1000,185.70808
+7061.500099,924,186.991367
+5629.528694,735,188.014673
+7974.521715,1000,189.412562
+8043.671947,1000,190.806722
+7843.185138,994,192.178238
+7889.325323,1000,193.558639
+5534.878237,694,194.519561
+8074.045734,1000,195.901475
+4227.32051,543,196.655645
+8079.992894,1000,198.041193
+8091.688938,1000,199.439367
+8018.843051,1000,200.60383
+8079.905703,1000,201.725068
+8090.082906,1000,202.949079
+7999.612591,1000,204.328829
+8111.457637,1000,205.726873
diff --git a/logs/benchmark/tqc-Swimmer-v3/0.monitor.csv b/logs/benchmark/tqc-Swimmer-v3/0.monitor.csv
new file mode 100644
index 000000000..d7cba6e05
--- /dev/null
+++ b/logs/benchmark/tqc-Swimmer-v3/0.monitor.csv
@@ -0,0 +1,152 @@
+#{"t_start": 1635520213.6787107, "env_id": "Swimmer-v3"}
+r,l,t
+339.704525,1000,0.656019
+340.920584,1000,1.292514
+340.61682,1000,1.931788
+339.719078,1000,2.582214
+340.239328,1000,3.240417
+338.760491,1000,3.890345
+338.861221,1000,4.527725
+340.733655,1000,5.182431
+340.987331,1000,5.834769
+340.001075,1000,6.485663
+338.078409,1000,7.125048
+340.833256,1000,7.7774
+341.243228,1000,8.431881
+341.022868,1000,9.07746
+337.048048,1000,9.725487
+340.034504,1000,10.362979
+340.219137,1000,11.010483
+343.494797,1000,11.661463
+340.107313,1000,12.312516
+338.686568,1000,12.952533
+339.216484,1000,13.598711
+340.501221,1000,14.246438
+340.079585,1000,14.892798
+339.997192,1000,15.539901
+340.144739,1000,16.175275
+338.621794,1000,16.820345
+340.126661,1000,17.467686
+339.360154,1000,18.115562
+340.075502,1000,18.766376
+337.886429,1000,19.414195
+340.477407,1000,20.064856
+336.978552,1000,20.706122
+338.875227,1000,21.350773
+337.894396,1000,21.995255
+338.267948,1000,22.63968
+337.982914,1000,23.287703
+339.715304,1000,23.933161
+338.595989,1000,24.580248
+337.885833,1000,25.22817
+339.355769,1000,25.874181
+338.08875,1000,26.521915
+339.835957,1000,27.159792
+336.914019,1000,27.816386
+338.951695,1000,28.458207
+340.392355,1000,29.119091
+338.483415,1000,29.7553
+339.350702,1000,30.40123
+339.684779,1000,31.05004
+338.423293,1000,31.696265
+339.733623,1000,32.344393
+338.318244,1000,33.001592
+340.764995,1000,33.64474
+339.19359,1000,34.289434
+338.593334,1000,34.934269
+340.942486,1000,35.592097
+340.17316,1000,36.259211
+337.647758,1000,36.911052
+338.400646,1000,37.56419
+336.512724,1000,38.229111
+339.182615,1000,38.871707
+337.747266,1000,39.517359
+339.786163,1000,40.160728
+338.781046,1000,40.798184
+337.860217,1000,41.443889
+337.425911,1000,42.088646
+341.499117,1000,42.734946
+339.278337,1000,43.38132
+337.257005,1000,44.029842
+340.33294,1000,44.678912
+344.126499,1000,45.324063
+341.268077,1000,45.971306
+339.267486,1000,46.610621
+339.167546,1000,47.271691
+341.427853,1000,47.917441
+339.099635,1000,48.559796
+338.510212,1000,49.217607
+340.416555,1000,49.866024
+341.612665,1000,50.51829
+340.66108,1000,51.159971
+340.027186,1000,51.804467
+337.715366,1000,52.45233
+337.849216,1000,53.09846
+340.263863,1000,53.74347
+339.117614,1000,54.387378
+338.506886,1000,55.038113
+338.113889,1000,55.681837
+338.926911,1000,56.332048
+338.198536,1000,56.986665
+338.819372,1000,57.626685
+343.158296,1000,58.276741
+339.814723,1000,58.926339
+338.545918,1000,59.570472
+338.618989,1000,60.218882
+338.085244,1000,60.872674
+341.048935,1000,61.518309
+340.108426,1000,62.165694
+338.473547,1000,62.81012
+339.642958,1000,63.464669
+339.588795,1000,64.129526
+337.033032,1000,64.767526
+344.779796,1000,65.421099
+339.6424,1000,66.082355
+340.763807,1000,66.736104
+337.833639,1000,67.388196
+340.349732,1000,68.046243
+340.453125,1000,68.698389
+343.822102,1000,69.342707
+338.728552,1000,69.998209
+338.418623,1000,70.655675
+338.015934,1000,71.303368
+339.575131,1000,71.949526
+344.844971,1000,72.590318
+337.681243,1000,73.233643
+338.438761,1000,73.876229
+338.278767,1000,74.538148
+340.183888,1000,75.182518
+339.676421,1000,75.826741
+338.518092,1000,76.47313
+339.103418,1000,77.124239
+341.272023,1000,77.767996
+339.198562,1000,78.412756
+337.680646,1000,79.05841
+337.955362,1000,79.706059
+338.661022,1000,80.349332
+338.712666,1000,80.991192
+337.043867,1000,81.642758
+340.697279,1000,82.290753
+341.110325,1000,82.935584
+340.054736,1000,83.584702
+339.650687,1000,84.229111
+338.651965,1000,84.874009
+337.256805,1000,85.524997
+339.093129,1000,86.166047
+340.254517,1000,86.811029
+337.641425,1000,87.452493
+340.16177,1000,88.098247
+338.249355,1000,88.741568
+338.717443,1000,89.389659
+340.033707,1000,90.038833
+339.690869,1000,90.677379
+339.304048,1000,91.326945
+340.43305,1000,91.972727
+336.565726,1000,92.620342
+340.419783,1000,93.268865
+339.132569,1000,93.917203
+339.772248,1000,94.565035
+338.825575,1000,95.211792
+340.115401,1000,95.877755
+339.145544,1000,96.56019
+338.753283,1000,97.206737
diff --git a/logs/benchmark/tqc-Walker2d-v3/0.monitor.csv b/logs/benchmark/tqc-Walker2d-v3/0.monitor.csv
new file mode 100644
index 000000000..b6df6bbd0
--- /dev/null
+++ b/logs/benchmark/tqc-Walker2d-v3/0.monitor.csv
@@ -0,0 +1,154 @@
+#{"t_start": 1635520425.255643, "env_id": "Walker2d-v3"}
+r,l,t
+4442.264149,1000,1.059754
+4542.608714,1000,2.144004
+4512.607985,1000,3.179214
+4504.605908,1000,4.28166
+4431.478183,1000,5.596047
+4492.012873,1000,6.924724
+4280.184618,1000,8.394533
+4497.463834,1000,9.966327
+4510.457687,1000,11.147074
+4458.265821,1000,12.066937
+4497.454299,1000,13.02112
+4302.195818,1000,13.925668
+4398.246774,1000,14.78906
+4469.658082,1000,15.68192
+4468.481716,1000,16.560531
+4492.883478,1000,17.478317
+4456.035014,1000,18.376341
+4467.439538,1000,19.311914
+4431.247244,1000,20.187187
+4531.362431,1000,21.058609
+4415.447355,1000,21.958295
+4441.38045,1000,22.839588
+4466.58657,1000,23.715653
+4488.39,1000,24.577008
+4423.479017,1000,25.44338
+4616.744965,1000,26.295029
+4435.49205,1000,27.165506
+4425.053325,1000,28.122981
+1427.289181,399,28.555349
+4372.322723,1000,29.461657
+4385.53961,1000,30.333291
+4409.610121,1000,31.190455
+4417.62247,1000,32.130104
+4436.953354,1000,33.204299
+4476.947994,1000,34.299675
+4525.089133,1000,35.478183
+4460.562914,1000,36.72362
+4494.603212,1000,37.579745
+4285.382236,1000,38.384158
+4465.133309,1000,39.207988
+4539.84139,1000,40.080816
+4420.112839,1000,41.075914
+4402.420689,1000,41.914884
+4514.39929,1000,42.835467
+4468.929659,1000,44.410455
+1474.353237,447,45.321378
+4507.655891,1000,46.973942
+4462.185519,1000,48.351296
+4413.857608,1000,49.508611
+4487.923136,1000,50.9773
+4467.973784,1000,52.322625
+4445.270727,1000,53.434761
+4509.118493,1000,54.551867
+4506.716544,1000,55.69226
+4442.353719,1000,57.012844
+4497.767296,1000,58.219556
+4525.669206,1000,59.198278
+4602.370792,1000,60.232181
+1168.780617,362,60.695582
+4513.927223,1000,61.694278
+4414.726847,1000,62.902152
+4599.042528,1000,64.057207
+4498.190392,1000,65.23186
+4429.313639,1000,67.086337
+4502.571383,1000,68.796006
+4381.106683,1000,70.293839
+4501.034621,1000,72.239305
+4514.274073,1000,74.477244
+4602.369345,1000,76.502607
+4460.706776,1000,78.599248
+4518.431643,1000,80.358397
+4392.01215,1000,81.950115
+4196.51666,1000,83.540807
+4429.922947,1000,84.718932
+4400.571533,1000,85.809177
+4500.544119,1000,86.915489
+4475.033527,1000,88.006912
+4405.167465,1000,89.09914
+4469.110212,1000,90.185548
+4509.601644,1000,91.479958
+4483.062434,1000,92.572694
+4493.174621,1000,93.653375
+4400.963556,1000,94.758333
+4446.159459,1000,95.850652
+4486.42202,1000,96.953254
+4487.789241,1000,98.034734
+4385.150742,1000,99.084483
+4124.053272,1000,100.15711
+4512.805789,1000,101.247666
+4436.105112,1000,102.49271
+4273.197802,1000,103.934765
+4556.000243,1000,105.758611
+4361.786846,1000,107.476212
+4603.281898,1000,108.735101
+4502.048201,1000,109.970357
+4467.476333,1000,111.222079
+4415.305514,1000,112.818039
+4461.417116,1000,114.199019
+4510.090345,1000,115.850142
+4412.627944,1000,117.313183
+4391.311043,1000,118.660731
+4461.081558,1000,120.207889
+4389.811502,1000,121.843008
+4476.274731,1000,123.224241
+4419.75939,1000,124.879091
+4539.692325,1000,126.239017
+4424.297314,1000,127.523315
+4431.986001,1000,128.739518
+4567.96953,1000,129.927734
+4432.852459,1000,131.304051
+4496.414587,1000,132.633009
+4412.390476,1000,133.582246
+4450.164474,1000,134.473694
+4399.862418,1000,135.727114
+4457.926102,1000,137.045931
+4539.058758,1000,138.049587
+4468.690172,1000,138.893485
+4559.232717,1000,139.730619
+4502.582618,1000,140.522552
+4426.517989,1000,141.339646
+4502.595669,1000,142.142529
+4580.883032,1000,142.965874
+4530.972211,1000,143.796237
+4509.440038,1000,144.602551
+4646.985095,1000,145.428639
+4569.048817,1000,146.277569
+4444.709489,1000,147.130211
+4469.148454,1000,147.932139
+4461.519102,1000,148.741906
+4436.817486,1000,149.602266
+4148.112699,1000,150.428048
+4596.802826,1000,151.2355
+4449.086702,1000,152.22748
+4490.743391,1000,153.162584
+4501.211089,1000,154.034612
+4466.611291,1000,154.948788
+4523.690894,1000,155.753219
+4456.958811,1000,156.558347
+4489.876482,1000,157.355024
+4430.926954,1000,158.138672
+4400.561192,1000,158.941687
+4537.969915,1000,159.834009
+4456.593106,1000,160.671252
+4415.831542,1000,161.478463
+4456.87806,1000,162.262658
+4507.464381,1000,163.077074
+4436.045271,1000,163.940326
+4429.715771,1000,164.783452
+1422.725116,398,165.103478
+4506.386056,1000,165.92082
+4497.402696,1000,166.743915
+4400.546846,1000,167.555964
diff --git a/rl-trained-agents b/rl-trained-agents
index 3dd2af4ce..493e183da 160000
--- a/rl-trained-agents
+++ b/rl-trained-agents
@@ -1,1 +1,1 @@
-Subproject commit 3dd2af4cee930750016cf943dc6393bada57b89c
+Subproject commit 493e183da3525faebdc72fa029bd48f923891e95
diff --git a/scripts/create_mujoco_jobs.py b/scripts/create_mujoco_jobs.py
new file mode 100644
index 000000000..dc53f3187
--- /dev/null
+++ b/scripts/create_mujoco_jobs.py
@@ -0,0 +1,55 @@
+import os
+import subprocess
+import time
+
+import numpy as np
+
+ALGOS = ["sac", "td3", "tqc"]
+# "Humanoid-v3",
+ENVS = ["HalfCheetah-v3", "Ant-v3", "Hopper-v3", "Walker2d-v3", "Swimmer-v3"]
+N_SEEDS = 1
+EVAL_FREQ = 25000
+N_EVAL_EPISODES = 20
+N_EVAL_ENVS = 5
+np.random.seed(8)
+SEEDS = np.random.randint(2 ** 20, size=(N_SEEDS,))
+# N_TIMESTEPS = int(1e6)
+
+os.makedirs(os.path.join("logs", "slurm"), exist_ok=True)
+log_folder = "logs/"
+
+
+for algo in ALGOS:
+    for env_id in ENVS:
+        for seed in SEEDS:
+            args = [
+                "--algo",
+                algo,
+                "--env",
+                env_id,
+                # "--hyperparams",
+                # "use_sde:False",
+                "--eval-episodes",
+                N_EVAL_EPISODES,
+                "--eval-freq",
+                EVAL_FREQ,
+                "--n-eval-envs",
+                N_EVAL_ENVS,
+                "-f",
+                log_folder,
+                "--seed",
+                seed,
+                "--log-interval",
+                10,
+                "--num-threads",
+                2,
+                # "-n",
+                # N_TIMESTEPS,
+                "-uuid",
+            ]
+            args = list(map(str, args))
+
+            command = " ".join(["python", "-u", "train.py"] + args)
+
+            ok = subprocess.call(["sbatch", "cluster_torchy.sh", algo, env_id, "ablation", command])
+            time.sleep(0.05)
diff --git a/utils/benchmark.py b/utils/benchmark.py
index fc7e2fdd4..9c7e8d597 100644
--- a/utils/benchmark.py
+++ b/utils/benchmark.py
@@ -19,7 +19,7 @@
 parser.add_argument("--verbose", help="Verbose mode (0: no output, 1: INFO)", default=1, type=int)
 parser.add_argument("--seed", help="Random generator seed", type=int, default=0)
 parser.add_argument("--test-mode", action="store_true", default=False, help="Do only one experiment (useful for testing)")
-parser.add_argument("--with-mujoco", action="store_true", default=False, help="Run also MuJoCo envs (require a licence)")
+parser.add_argument("--with-mujoco", action="store_true", default=False, help="Run also MuJoCo envs")
 parser.add_argument("--num-threads", help="Number of threads for PyTorch", default=2, type=int)
 args = parser.parse_args()
 
diff --git a/version.txt b/version.txt
index f0bb29e76..c5813fb1b 100644
--- a/version.txt
+++ b/version.txt
@@ -1,1 +1,1 @@
-1.3.0
+1.3.1a2


Test patch:
diff --git a/tests/test_enjoy.py b/tests/test_enjoy.py
index 6eea0eaeb..d6ab4b948 100644
--- a/tests/test_enjoy.py
+++ b/tests/test_enjoy.py
@@ -27,7 +27,7 @@ def test_trained_agents(trained_model):
         return
 
     # Skip mujoco envs
-    if "Fetch" in trained_model:
+    if "Fetch" in trained_model or "-v3" in trained_model:
         return
 
     if "-MiniGrid-" in trained_model: