Repository: DLR-RM/rl-baselines3-zoo (language: python)

Issue:
Upgrade to Gym 0.21
## Description

Upgrade to Gym version 0.21 now that SB3 master has shifted. Mostly just requires changes in environment names.

I have *not* regenerated the trained agents or regenerated the benchmarks. I couldn't easily figure out what pipeline is used for this? If you can point me to the relevant scripts I can do this.

`requirements.txt` is currently pointing to SB3 master, should make SB3 1.4.1 release first, then update requirements here then merge.

## Motivation and Context

Discussed in https://github.com/DLR-RM/stable-baselines3/pull/734 https://github.com/DLR-RM/stable-baselines3/issues/674

## Types of changes

- [X] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
- [ ] Documentation (update in the documentation)

## Checklist:
<!--- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!--- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [X] I've read the [CONTRIBUTION](https://github.com/DLR-RM/stable-baselines3/blob/master/CONTRIBUTING.md) guide (**required**)
- [ ] I have updated the changelog accordingly (**required**).
I can't see the changelog?

- [ ] My change requires a change to the documentation.
- [ ] I have updated the tests accordingly (*required for a bug fix or a new feature*).
- [ ] I have updated the documentation accordingly.
- [X] I have reformatted the code using `make format` (**required**)
- [X] I have checked the codestyle using `make check-codestyle` and `make lint` (**required**)
- [X] I have ensured `make pytest` and `make type` both pass. (**required**)

Note: we are using a maximum length of 127 characters per line

<!--- This Template is an edited version of the one from https://github.com/evilsocket/pwnagotchi/ -->



Discussion / hints:

Fix patch:
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index cbba279f0..24390204e 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -35,8 +35,8 @@ jobs:
         pip install -r requirements.txt
         # Use headless version
         pip install opencv-python-headless
-        # install parking-env to test HER
-        pip install git+https://github.com/eleurent/highway-env
+        # install parking-env to test HER (pinned so it works with gym 0.21)
+        pip install git+https://github.com/eleurent/highway-env@1a04c6a98be64632cf9683625022023e70ff1ab1
     - name: Type check
       run: |
         make type
diff --git a/.github/workflows/trained_agents.yml b/.github/workflows/trained_agents.yml
index 425457a87..39a970f85 100644
--- a/.github/workflows/trained_agents.yml
+++ b/.github/workflows/trained_agents.yml
@@ -34,8 +34,8 @@ jobs:
         pip install -r requirements.txt
         # Use headless version
         pip install opencv-python-headless
-        # install parking-env to test HER
-        pip install git+https://github.com/eleurent/highway-env
+        # install parking-env to test HER (pinned so it works with gym 0.21)
+        pip install git+https://github.com/eleurent/highway-env@1a04c6a98be64632cf9683625022023e70ff1ab1
         # Add support for pickle5 protocol
         pip install pickle5
     - name: Check trained agents
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index d0f7861e1..6b6dfe1b3 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -16,7 +16,7 @@ pytest:
 check-trained-agents:
   script:
   # MKL_THREADING_LAYER=GNU to avoid MKL_THREADING_LAYER=INTEL incompatibility error
-  - pip install pickle5  # Add support for pickle6 protocol
+  - pip install pickle5  # Add support for pickle5 protocol
   - MKL_THREADING_LAYER=GNU make check-trained-agents
 
 lint:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 43947c057..17e28ffe6 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,3 +1,19 @@
+## Release 1.4.1a1 (WIP)
+
+### Breaking Changes
+- Upgrade to Stable-Baselines3 (SB3) >= 1.4.1a1
+- Upgrade to sb3-contrib >= 1.4.1a1
+- Upgraded to gym 0.21
+
+### New Features
+
+### Bug fixes
+
+### Documentation
+
+### Other
+
+
 ## Release 1.4.0 (2022-01-19)
 
 ### Breaking Changes
diff --git a/Makefile b/Makefile
index 83946b0be..bb8788e6c 100644
--- a/Makefile
+++ b/Makefile
@@ -6,7 +6,7 @@ pytest:
 
 # check all trained agents (slow)
 check-trained-agents:
-	python -m pytest -v tests/test_enjoy.py -k trained_agent
+	python -m pytest -v tests/test_enjoy.py -k trained_agent --color=yes
 
 # Type check
 type:
diff --git a/README.md b/README.md
index 124742dee..e1dc5f2f4 100644
--- a/README.md
+++ b/README.md
@@ -55,7 +55,7 @@ python train.py --algo a2c --env BreakoutNoFrameskip-v4 -i rl-trained-agents/a2c
 
 When using off-policy algorithms, you can also save the replay buffer after training:
 ```
-python train.py --algo sac --env Pendulum-v0 --save-replay-buffer
+python train.py --algo sac --env Pendulum-v1 --save-replay-buffer
 ```
 It will be automatically loaded if present when continuing training.
 
@@ -320,13 +320,13 @@ Additional Atari Games (to be completed):
 
 ### Classic Control Environments
 
-|  RL Algo |  CartPole-v1 | MountainCar-v0 | Acrobot-v1 |  Pendulum-v0 | MountainCarContinuous-v0 |
-|----------|--------------|----------------|------------|--------------|--------------------------|
-| ARS      | :heavy_check_mark: | :heavy_check_mark:  | :heavy_check_mark: | :heavy_check_mark:  | :heavy_check_mark: |
-| A2C      | :heavy_check_mark: | :heavy_check_mark:  | :heavy_check_mark: | :heavy_check_mark:  | :heavy_check_mark: |
-| PPO      | :heavy_check_mark: | :heavy_check_mark:  | :heavy_check_mark: | :heavy_check_mark:  | :heavy_check_mark: |
-| DQN      | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | N/A | N/A |
-| QR-DQN   | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | N/A | N/A |
+|  RL Algo |  CartPole-v1 | MountainCar-v0 | Acrobot-v1 | Pendulum-v1 | MountainCarContinuous-v0 |
+|----------|--------------|----------------|------------|--------------------|--------------------------|
+| ARS      | :heavy_check_mark: | :heavy_check_mark:  | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |
+| A2C      | :heavy_check_mark: | :heavy_check_mark:  | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |
+| PPO      | :heavy_check_mark: | :heavy_check_mark:  | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |
+| DQN      | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | N/A                | N/A |
+| QR-DQN   | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | N/A                | N/A |
 | DDPG     |  N/A |  N/A  | N/A | :heavy_check_mark: | :heavy_check_mark: |
 | SAC      |  N/A |  N/A  | N/A | :heavy_check_mark: | :heavy_check_mark: |
 | TD3      |  N/A |  N/A  | N/A | :heavy_check_mark: | :heavy_check_mark: |
diff --git a/benchmark.md b/benchmark.md
index 0642269c3..d50f5c1d7 100644
--- a/benchmark.md
+++ b/benchmark.md
@@ -34,7 +34,7 @@ and also allow users to have access to pretrained agents.*
 |a2c  |LunarLanderContinuous-v2   |     84.225|   145.906|5M         |        149305|          256|
 |a2c  |MountainCar-v0             |   -111.263|    24.087|1M         |        149982|         1348|
 |a2c  |MountainCarContinuous-v0   |     91.166|     0.255|100k       |        149923|         1659|
-|a2c  |Pendulum-v0                |   -162.965|   103.210|1M         |        150000|          750|
+|a2c  |Pendulum-v1                |   -162.965|   103.210|1M         |        150000|          750|
 |a2c  |PongNoFrameskip-v4         |     17.292|     3.214|10M        |        594910|           65|
 |a2c  |QbertNoFrameskip-v4        |   3882.345|  1223.327|10M        |        610670|          194|
 |a2c  |ReacherBulletEnv-v0        |     14.968|    10.978|2M         |        150000|         1000|
@@ -51,7 +51,7 @@ and also allow users to have access to pretrained agents.*
 |ars  |LunarLanderContinuous-v2   |    167.959|   147.071|2M         |        149883|          562|
 |ars  |MountainCar-v0             |   -122.000|    33.456|500k       |        149938|         1229|
 |ars  |MountainCarContinuous-v0   |     96.672|     0.784|500k       |        149990|          621|
-|ars  |Pendulum-v0                |   -212.540|   160.444|2M         |        150000|          750|
+|ars  |Pendulum-v1                |   -212.540|   160.444|2M         |        150000|          750|
 |ars  |Swimmer-v3                 |    355.267|    12.796|2M         |        150000|          150|
 |ars  |Walker2d-v3                |   2993.582|   166.289|75M        |        149821|          152|
 |ddpg |AntBulletEnv-v0            |   2399.147|    75.410|1M         |        150000|          150|
@@ -60,7 +60,7 @@ and also allow users to have access to pretrained agents.*
 |ddpg |HopperBulletEnv-v0         |   1157.065|   448.695|1M         |        149565|          346|
 |ddpg |LunarLanderContinuous-v2   |    230.217|    92.372|300k       |        149862|          556|
 |ddpg |MountainCarContinuous-v0   |     93.512|     0.048|300k       |        149965|         2260|
-|ddpg |Pendulum-v0                |   -152.099|    94.282|20k        |        150000|          750|
+|ddpg |Pendulum-v1                |   -152.099|    94.282|20k        |        150000|          750|
 |ddpg |ReacherBulletEnv-v0        |     15.582|     9.606|300k       |        150000|         1000|
 |ddpg |Walker2DBulletEnv-v0       |   1387.591|   736.955|1M         |        149051|          208|
 |dqn  |Acrobot-v1                 |    -76.639|    11.752|100k       |        149998|         1932|
@@ -94,7 +94,7 @@ and also allow users to have access to pretrained agents.*
 |ppo  |LunarLanderContinuous-v2   |    270.863|    32.072|1M         |        149956|          526|
 |ppo  |MountainCar-v0             |   -110.423|    19.473|1M         |        149954|         1358|
 |ppo  |MountainCarContinuous-v0   |     88.343|     2.572|20k        |        149983|          633|
-|ppo  |Pendulum-v0                |   -172.225|   104.159|100k       |        150000|          750|
+|ppo  |Pendulum-v1                |   -172.225|   104.159|100k       |        150000|          750|
 |ppo  |PongNoFrameskip-v4         |     20.989|     0.105|10M        |        599902|           90|
 |ppo  |QbertNoFrameskip-v4        |  15627.108|  3313.538|10M        |        600248|           83|
 |ppo  |ReacherBulletEnv-v0        |     17.091|    11.048|1M         |        150000|         1000|
@@ -128,7 +128,7 @@ and also allow users to have access to pretrained agents.*
 |sac  |Humanoid-v3                |   6232.287|   279.885|2M         |        149460|          150|
 |sac  |LunarLanderContinuous-v2   |    260.390|    65.467|500k       |        149634|          672|
 |sac  |MountainCarContinuous-v0   |     94.679|     1.134|50k        |        149966|         1443|
-|sac  |Pendulum-v0                |   -156.995|    88.714|20k        |        150000|          750|
+|sac  |Pendulum-v1                |   -156.995|    88.714|20k        |        150000|          750|
 |sac  |ReacherBulletEnv-v0        |     18.062|     9.729|300k       |        150000|         1000|
 |sac  |Swimmer-v3                 |    345.568|     3.084|1M         |        150000|          150|
 |sac  |Walker2DBulletEnv-v0       |   2292.266|    13.970|1M         |        149983|          150|
@@ -144,7 +144,7 @@ and also allow users to have access to pretrained agents.*
 |td3  |Humanoid-v3                |   5566.687|    14.544|2M         |        150000|          150|
 |td3  |LunarLanderContinuous-v2   |    207.451|    67.562|300k       |        149488|          337|
 |td3  |MountainCarContinuous-v0   |     93.483|     0.075|300k       |        149976|         2275|
-|td3  |Pendulum-v0                |   -151.855|    90.227|20k        |        150000|          750|
+|td3  |Pendulum-v1                |   -151.855|    90.227|20k        |        150000|          750|
 |td3  |ReacherBulletEnv-v0        |     17.114|     9.750|300k       |        150000|         1000|
 |td3  |Swimmer-v3                 |    359.127|     1.244|1M         |        150000|          150|
 |td3  |Walker2DBulletEnv-v0       |   2213.672|   230.558|1M         |        149800|          152|
@@ -169,7 +169,7 @@ and also allow users to have access to pretrained agents.*
 |tqc  |PandaReach-v1              |     -1.768|     0.858|20k        |        150000|         3000|
 |tqc  |PandaSlide-v1              |    -27.497|     9.868|3M         |        150000|         3000|
 |tqc  |PandaStack-v1              |    -96.915|    17.240|1M         |        150000|         1500|
-|tqc  |Pendulum-v0                |   -151.340|    87.893|20k        |        150000|          750|
+|tqc  |Pendulum-v1                |   -151.340|    87.893|20k        |        150000|          750|
 |tqc  |ReacherBulletEnv-v0        |     18.255|     9.543|300k       |        150000|         1000|
 |tqc  |Swimmer-v3                 |    339.423|     1.486|1M         |        150000|          150|
 |tqc  |Walker2DBulletEnv-v0       |   2508.934|   614.624|1M         |        149572|          159|
@@ -188,7 +188,7 @@ and also allow users to have access to pretrained agents.*
 |trpo |LunarLanderContinuous-v2   |    262.387|    21.428|200k       |        149925|          501|
 |trpo |MountainCar-v0             |   -107.278|    13.231|100k       |        149974|         1398|
 |trpo |MountainCarContinuous-v0   |     92.489|     0.355|50k        |        149971|         1732|
-|trpo |Pendulum-v0                |   -174.631|   127.577|100k       |        150000|          750|
+|trpo |Pendulum-v1                |   -174.631|   127.577|100k       |        150000|          750|
 |trpo |ReacherBulletEnv-v0        |     14.741|    11.559|300k       |        150000|         1000|
 |trpo |Swimmer-v3                 |    365.663|     2.087|1M         |        150000|          150|
 |trpo |Walker2DBulletEnv-v0       |   1483.467|   823.468|2M         |        149860|          197|
diff --git a/docker/Dockerfile b/docker/Dockerfile
index a2e1dcae2..a99e9a7b1 100644
--- a/docker/Dockerfile
+++ b/docker/Dockerfile
@@ -21,11 +21,9 @@ RUN \
     mkdir -p ${CODE_DIR}/rl_zoo && \
     pip uninstall -y stable-baselines3 && \
     pip install -r /tmp/requirements.txt && \
-    pip install git+https://github.com/eleurent/highway-env && \
+    pip install git+https://github.com/eleurent/highway-env@1a04c6a98be64632cf9683625022023e70ff1ab1 && \
     rm -rf $HOME/.cache/pip
 
-RUN pip install gym==0.19.0
-
 ENV PATH=$VENV/bin:$PATH
 
 COPY docker/entrypoint.sh /tmp/
diff --git a/hyperparams/a2c.yml b/hyperparams/a2c.yml
index a534a51d1..f51b0fb09 100644
--- a/hyperparams/a2c.yml
+++ b/hyperparams/a2c.yml
@@ -39,7 +39,7 @@ Acrobot-v1:
   ent_coef: .0
 
 # Tuned
-Pendulum-v0:
+Pendulum-v1:
   normalize: True
   n_envs: 8
   n_timesteps: !!float 1e6
diff --git a/hyperparams/ars.yml b/hyperparams/ars.yml
index 13897f4ec..e58d4fa3c 100644
--- a/hyperparams/ars.yml
+++ b/hyperparams/ars.yml
@@ -6,7 +6,7 @@ CartPole-v1:
   n_delta: 2
 
 # Tuned
-Pendulum-v0: &pendulum-params
+Pendulum-v1: &pendulum-params
   n_envs: 1
   n_timesteps: !!float 2e6
   policy: 'MlpPolicy'
diff --git a/hyperparams/ddpg.yml b/hyperparams/ddpg.yml
index e86be61db..14a53cfca 100644
--- a/hyperparams/ddpg.yml
+++ b/hyperparams/ddpg.yml
@@ -5,7 +5,7 @@ MountainCarContinuous-v0:
   noise_type: 'ornstein-uhlenbeck'
   noise_std: 0.5
 
-Pendulum-v0:
+Pendulum-v1:
   n_timesteps: 20000
   policy: 'MlpPolicy'
   gamma: 0.98
diff --git a/hyperparams/ppo.yml b/hyperparams/ppo.yml
index a51cc1771..2a88c30d8 100644
--- a/hyperparams/ppo.yml
+++ b/hyperparams/ppo.yml
@@ -14,7 +14,7 @@ atari:
   ent_coef: 0.01
 
 # Tuned
-Pendulum-v0:
+Pendulum-v1:
   n_envs: 4
   n_timesteps: !!float 1e5
   policy: 'MlpPolicy'
diff --git a/hyperparams/sac.yml b/hyperparams/sac.yml
index 6b1cdedc5..8aeaef616 100644
--- a/hyperparams/sac.yml
+++ b/hyperparams/sac.yml
@@ -14,7 +14,7 @@ MountainCarContinuous-v0:
   use_sde: True
   policy_kwargs: "dict(log_std_init=-3.67, net_arch=[64, 64])"
 
-Pendulum-v0:
+Pendulum-v1:
   # callback:
   #   - utils.callbacks.ParallelTrainCallback
   n_timesteps: 20000
diff --git a/hyperparams/td3.yml b/hyperparams/td3.yml
index e761aab42..9b941516b 100644
--- a/hyperparams/td3.yml
+++ b/hyperparams/td3.yml
@@ -5,7 +5,7 @@ MountainCarContinuous-v0:
   noise_type: 'ornstein-uhlenbeck'
   noise_std: 0.5
 
-Pendulum-v0:
+Pendulum-v1:
   n_timesteps: 20000
   policy: 'MlpPolicy'
   gamma: 0.98
diff --git a/hyperparams/tqc.yml b/hyperparams/tqc.yml
index 73935304b..54e2f3a4d 100644
--- a/hyperparams/tqc.yml
+++ b/hyperparams/tqc.yml
@@ -14,7 +14,7 @@ MountainCarContinuous-v0:
   use_sde: True
   policy_kwargs: "dict(log_std_init=-3.67, net_arch=[64, 64])"
 
-Pendulum-v0:
+Pendulum-v1:
   n_timesteps: 20000
   policy: 'MlpPolicy'
   learning_rate: !!float 1e-3
diff --git a/hyperparams/trpo.yml b/hyperparams/trpo.yml
index 11e8b894d..c78263338 100644
--- a/hyperparams/trpo.yml
+++ b/hyperparams/trpo.yml
@@ -12,7 +12,7 @@ CartPole-v1:
   n_critic_updates: 20
 
 # Tuned
-Pendulum-v0:
+Pendulum-v1:
   n_envs: 2
   n_timesteps: !!float 1e5
   policy: 'MlpPolicy'
diff --git a/logs/benchmark/a2c-Pendulum-v0/0.monitor.csv b/logs/benchmark/a2c-Pendulum-v1/0.monitor.csv
similarity index 100%
rename from logs/benchmark/a2c-Pendulum-v0/0.monitor.csv
rename to logs/benchmark/a2c-Pendulum-v1/0.monitor.csv
diff --git a/logs/benchmark/ars-Pendulum-v0/0.monitor.csv b/logs/benchmark/ars-Pendulum-v1/0.monitor.csv
similarity index 100%
rename from logs/benchmark/ars-Pendulum-v0/0.monitor.csv
rename to logs/benchmark/ars-Pendulum-v1/0.monitor.csv
diff --git a/logs/benchmark/ddpg-Pendulum-v0/0.monitor.csv b/logs/benchmark/ddpg-Pendulum-v1/0.monitor.csv
similarity index 100%
rename from logs/benchmark/ddpg-Pendulum-v0/0.monitor.csv
rename to logs/benchmark/ddpg-Pendulum-v1/0.monitor.csv
diff --git a/logs/benchmark/ppo-Pendulum-v0/0.monitor.csv b/logs/benchmark/ppo-Pendulum-v1/0.monitor.csv
similarity index 100%
rename from logs/benchmark/ppo-Pendulum-v0/0.monitor.csv
rename to logs/benchmark/ppo-Pendulum-v1/0.monitor.csv
diff --git a/logs/benchmark/sac-Pendulum-v0/0.monitor.csv b/logs/benchmark/sac-Pendulum-v1/0.monitor.csv
similarity index 100%
rename from logs/benchmark/sac-Pendulum-v0/0.monitor.csv
rename to logs/benchmark/sac-Pendulum-v1/0.monitor.csv
diff --git a/logs/benchmark/td3-Pendulum-v0/0.monitor.csv b/logs/benchmark/td3-Pendulum-v1/0.monitor.csv
similarity index 100%
rename from logs/benchmark/td3-Pendulum-v0/0.monitor.csv
rename to logs/benchmark/td3-Pendulum-v1/0.monitor.csv
diff --git a/logs/benchmark/tqc-Pendulum-v0/0.monitor.csv b/logs/benchmark/tqc-Pendulum-v1/0.monitor.csv
similarity index 100%
rename from logs/benchmark/tqc-Pendulum-v0/0.monitor.csv
rename to logs/benchmark/tqc-Pendulum-v1/0.monitor.csv
diff --git a/logs/benchmark/trpo-Pendulum-v0/0.monitor.csv b/logs/benchmark/trpo-Pendulum-v1/0.monitor.csv
similarity index 100%
rename from logs/benchmark/trpo-Pendulum-v0/0.monitor.csv
rename to logs/benchmark/trpo-Pendulum-v1/0.monitor.csv
diff --git a/requirements.txt b/requirements.txt
index 99e3b7a69..2b6096d78 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,6 +1,6 @@
-gym>=0.17,<0.20
-stable-baselines3[extra,tests,docs]>=1.4.0
-sb3-contrib>=1.4.0
+gym==0.21
+stable-baselines3[extra,tests,docs]>=1.4.1a1
+sb3-contrib>=1.4.1a1
 box2d-py==2.3.8
 pybullet
 gym-minigrid
@@ -10,8 +10,6 @@ pytablewriter
 seaborn
 pyyaml>=5.1
 cloudpickle>=1.5.0
-# tmp fix: ROM missing in newest release
-atari-py==0.2.6
 plotly
 panda-gym==1.1.1 # tmp fix: until compatibility with panda-gym v2
 rliable>=1.0.5
diff --git a/rl-trained-agents b/rl-trained-agents
index 10a9c31e8..10d59c066 160000
--- a/rl-trained-agents
+++ b/rl-trained-agents
@@ -1,1 +1,1 @@
-Subproject commit 10a9c31e806820d59b20d8b85ca67090338ea912
+Subproject commit 10d59c06619c4a15de46c7db0f4f4076090533f6
diff --git a/scripts/build_docker.sh b/scripts/build_docker.sh
index 078b4f9e4..e074fc78a 100755
--- a/scripts/build_docker.sh
+++ b/scripts/build_docker.sh
@@ -3,7 +3,7 @@
 PARENT=stablebaselines/stable-baselines3
 
 TAG=stablebaselines/rl-baselines3-zoo
-VERSION=1.3.1a3
+VERSION=1.4.1a0
 
 if [[ ${USE_GPU} == "True" ]]; then
   PARENT="${PARENT}:${VERSION}"
diff --git a/scripts/create_mujoco_jobs.py b/scripts/create_mujoco_jobs.py
index dc53f3187..e7efd6cfb 100644
--- a/scripts/create_mujoco_jobs.py
+++ b/scripts/create_mujoco_jobs.py
@@ -12,7 +12,7 @@
 N_EVAL_EPISODES = 20
 N_EVAL_ENVS = 5
 np.random.seed(8)
-SEEDS = np.random.randint(2 ** 20, size=(N_SEEDS,))
+SEEDS = np.random.randint(2**20, size=(N_SEEDS,))
 # N_TIMESTEPS = int(1e6)
 
 os.makedirs(os.path.join("logs", "slurm"), exist_ok=True)
diff --git a/train.py b/train.py
index 49f749a3a..739d62394 100644
--- a/train.py
+++ b/train.py
@@ -135,7 +135,7 @@
     uuid_str = f"_{uuid.uuid4()}" if args.uuid else ""
     if args.seed < 0:
         # Seed but with a random one
-        args.seed = np.random.randint(2 ** 32 - 1, dtype="int64").item()
+        args.seed = np.random.randint(2**32 - 1, dtype="int64").item()
 
     set_random_seed(args.seed)
 
diff --git a/version.txt b/version.txt
index 88c5fb891..d012e1c67 100644
--- a/version.txt
+++ b/version.txt
@@ -1,1 +1,1 @@
-1.4.0
+1.4.1a1


Test patch:
diff --git a/tests/test_enjoy.py b/tests/test_enjoy.py
index d6ab4b948..c55e702b2 100644
--- a/tests/test_enjoy.py
+++ b/tests/test_enjoy.py
@@ -81,14 +81,14 @@ def test_load(tmp_path):
 
 
 def test_record_video(tmp_path):
-    args = ["-n", "100", "--algo", "sac", "--env", "Pendulum-v0", "-o", str(tmp_path)]
+    args = ["-n", "100", "--algo", "sac", "--env", "Pendulum-v1", "-o", str(tmp_path)]
 
     # Skip if no X-Server
     pytest.importorskip("pyglet.gl")
 
     return_code = subprocess.call(["python", "-m", "utils.record_video"] + args)
     _assert_eq(return_code, 0)
-    video_path = str(tmp_path / "final-model-sac-Pendulum-v0-step-0-to-step-100.mp4")
+    video_path = str(tmp_path / "final-model-sac-Pendulum-v1-step-0-to-step-100.mp4")
     # File is not empty
     assert os.stat(video_path).st_size != 0, "Recorded video is empty"
 
diff --git a/tests/test_hyperparams_opt.py b/tests/test_hyperparams_opt.py
index 45ca4b3fc..cd5444b70 100644
--- a/tests/test_hyperparams_opt.py
+++ b/tests/test_hyperparams_opt.py
@@ -23,13 +23,13 @@ def _assert_eq(left, right):
         experiments[f"{algo}-{env_id}"] = (algo, env_id)
 
 # Test for SAC
-experiments["sac-Pendulum-v0"] = ("sac", "Pendulum-v0")
+experiments["sac-Pendulum-v1"] = ("sac", "Pendulum-v1")
 # Test for TD3
-experiments["td3-Pendulum-v0"] = ("td3", "Pendulum-v0")
+experiments["td3-Pendulum-v1"] = ("td3", "Pendulum-v1")
 # Test for HER
 experiments["tqc-parking-v0"] = ("tqc", "parking-v0")
 # Test for TQC
-experiments["tqc-Pendulum-v0"] = ("tqc", "Pendulum-v0")
+experiments["tqc-Pendulum-v1"] = ("tqc", "Pendulum-v1")
 
 
 @pytest.mark.parametrize("sampler", ["random", "tpe"])
diff --git a/tests/test_train.py b/tests/test_train.py
index 71c1297ef..ae0f5eda2 100644
--- a/tests/test_train.py
+++ b/tests/test_train.py
@@ -23,11 +23,11 @@ def _assert_eq(left, right):
 # Test for vecnormalize and frame-stack
 experiments["ppo-BipedalWalkerHardcore-v3"] = ("ppo", "BipedalWalkerHardcore-v3")
 # Test for SAC
-experiments["sac-Pendulum-v0"] = ("sac", "Pendulum-v0")
+experiments["sac-Pendulum-v1"] = ("sac", "Pendulum-v1")
 # for TD3
-experiments["td3-Pendulum-v0"] = ("td3", "Pendulum-v0")
+experiments["td3-Pendulum-v1"] = ("td3", "Pendulum-v1")
 # for DDPG
-experiments["ddpg-Pendulum-v0"] = ("ddpg", "Pendulum-v0")
+experiments["ddpg-Pendulum-v1"] = ("ddpg", "Pendulum-v1")
 
 
 @pytest.mark.parametrize("experiment", experiments.keys())
@@ -59,7 +59,7 @@ def test_continue_training(tmp_path):
 
 
 def test_save_load_replay_buffer(tmp_path):
-    algo, env_id = "sac", "Pendulum-v0"
+    algo, env_id = "sac", "Pendulum-v1"
     args = [
         "-n",
         str(N_STEPS),
@@ -77,9 +77,9 @@ def test_save_load_replay_buffer(tmp_path):
     return_code = subprocess.call(["python", "train.py"] + args)
     _assert_eq(return_code, 0)
 
-    assert os.path.isfile(os.path.join(tmp_path, "sac/Pendulum-v0_1/replay_buffer.pkl"))
+    assert os.path.isfile(os.path.join(tmp_path, "sac/Pendulum-v1_1/replay_buffer.pkl"))
 
-    args = args + ["-i", os.path.join(tmp_path, "sac/Pendulum-v0_1/Pendulum-v0.zip")]
+    args = args + ["-i", os.path.join(tmp_path, "sac/Pendulum-v1_1/Pendulum-v1.zip")]
 
     return_code = subprocess.call(["python", "train.py"] + args)
     _assert_eq(return_code, 0)
@@ -92,7 +92,7 @@ def test_parallel_train(tmp_path):
         "--algo",
         "sac",
         "--env",
-        "Pendulum-v0",
+        "Pendulum-v1",
         "--log-folder",
         tmp_path,
         "-params",