chrisluo5311/a2c-PandaPushDense-v3-potentialshaping

A2C Agent playing PandaPushDense-v3 (with potential-based reward shaping)

An A2C agent trained on PandaPushDense-v3 from panda-gym using stable-baselines3.

Reward shaping

The default dense reward is -||block_pos - goal_pos||. It does not change until the gripper touches the block, so there is no signal guiding the arm toward it. During training, a potential-based shaping term (Ng et al., 1999) rewards each step by how much closer the gripper got to the block:

phi(s)  = -k * ||ee_pos - block_pos||
reward  = original_dense_reward + (gamma * phi(s') - phi(s))

• ee_pos = obs["observation"][0:3], block_pos = obs["observation"][6:9]
• k = 0.5, gamma = 0.99 (same as A2C's discount factor)
• phi(s) is re-initialised on every reset(), and phi(s') = 0 when the episode terminates (success), so the shaping does not change the optimal policy.
• The shaping is applied only during training. Evaluation uses the unmodified environment.

Training setup

Item	Value
Algorithm	A2C (MultiInputPolicy, SB3 default hyperparameters)
Parallel envs	4
Total timesteps	1,000,000
VecNormalize	norm_obs=True, norm_reward=True, clip_obs=10, clip_reward=10

Usage

import gymnasium as gym
import panda_gym
from huggingface_sb3 import load_from_hub
from stable_baselines3 import A2C
from stable_baselines3.common.vec_env import DummyVecEnv, VecNormalize

repo_id = "chrisluo5311/a2c-PandaPushDense-v3-potentialshaping"
model = A2C.load(load_from_hub(repo_id, "a2c-PandaPushDense-v3-potentialshaping.zip"))

env = DummyVecEnv([lambda: gym.make("PandaPushDense-v3", render_mode="rgb_array")])
env = VecNormalize.load(load_from_hub(repo_id, "vec_normalize.pkl"), env)
env.training = False      # do not update normalization stats
env.norm_reward = False   # report the raw reward

obs = env.reset()
for _ in range(1000):
    action, _ = model.predict(obs, deterministic=True)
    obs, reward, done, info = env.step(action)