arabellako22/ppo-pusher-v5-vecnormalize

PPO Pusher-v5 with VecNormalize

This repository contains the best evaluation checkpoint from a Stable-Baselines3 PPO training run on Gymnasium Pusher-v5.

The checkpoint passed both release gates:

1. trained mean reward > random-policy mean reward
2. Deterministic action replay produced the same episode reward in the isolated MuJoCo rendering process.

Final evaluation

Metric	Result
Trained PPO mean reward	-28.93
Trained PPO standard deviation	3.67
PPO evaluation episodes	30
Random mean reward	-148.67
Random standard deviation	7.58
Random evaluation episodes	30
Improvement over random	119.74
Success criterion passed	True
Replay reward verified	True
Rendering backend	egl
Replay frames	101

For Pusher-v5, rewards are negative; values closer to zero are better.

Training configuration

Item	Value
Device	cpu
Seed	42
Vectorized environments	4
Requested timesteps	1000000
Actual timesteps	1007616
Selected checkpoint	best_evaluation_checkpoint
Callback best reward	-25.909009299999997
Evaluation interval	50000
Learning-rate schedule	3e-4 to 3e-5 linear
Policy network	256 x 256 ReLU
Value network	256 x 256 ReLU
gSDE	True
Batch size	256
Target KL	0.03

Included files

File	Purpose
ppo-Pusher-v5.zip	Best PPO evaluation checkpoint
vec_normalize.pkl	Matching VecNormalize statistics
training_config.json	Complete training configuration
evaluation.json	Final evaluation and replay verification
evaluations.npz	EvalCallback history
replay.mp4	Deterministic policy replay
requirements.txt	Reproduction dependencies

Replay

Open or download replay.mp4

Load the model

from stable_baselines3 import PPO
from stable_baselines3.common.env_util import make_vec_env
from stable_baselines3.common.vec_env import VecNormalize

env = make_vec_env(
    "Pusher-v5",
    n_envs=1,
)
env = VecNormalize.load(
    "vec_normalize.pkl",
    env,
)
env.training = False
env.norm_reward = False

model = PPO.load(
    "ppo-Pusher-v5.zip",
    env=env,
    device="cpu",
)

obs = env.reset()
for step in range(100):
    action, state = model.predict(
        obs,
        deterministic=True,
    )
    obs, rewards, dones, infos = env.step(action)

env.close()

The VecNormalize statistics must be loaded with the PPO checkpoint. Raw observations will not reproduce the reported result.

Rendering architecture

The evaluation and action-generation process imports Stable-Baselines3 but performs no rendering. A separate MuJoCo-only process replays the deterministic actions using the same seed. This prevents native OpenGL conflicts in headless Colab runtimes.

Package versions

• Gymnasium: 1.3.0
• Stable-Baselines3: 2.9.0
• MuJoCo: 3.12.0
• NumPy: 2.1.3
• PyTorch: 2.11.0+cpu
• ImageIO: 2.37.4

Reusable safe rendering module

The safe_mujoco_render.py module packages the subprocess-isolated evaluation and rendering workflow for other Gymnasium MuJoCo environments.

Supported configuration:

• Stable-Baselines3 PPO checkpoints
• Matching VecNormalize statistics
• Continuous Box action spaces
• EGL with OSMesa fallback
• Optional MP4 output
• Deterministic replay-reward verification

Usage

from safe_mujoco_render import (
    evaluate_and_render_ppo_mujoco,
)

result = evaluate_and_render_ppo_mujoco(
    env_id="Pusher-v5",
    model_path="ppo-Pusher-v5.zip",
    vecnormalize_path="vec_normalize.pkl",
    video_path="replay_from_module.mp4",
    device="cpu",
    n_eval_episodes=30,
    n_random_episodes=30,
    eval_seed=1000,
    random_seed=2000,
    replay_seed=42,
    width=640,
    height=480,
    backends=("egl", "osmesa"),
)

print(result)

Set video_path=None to validate frame generation without writing a video file. The function raises an exception if evaluation and rendering replay rewards do not match.