chrisluo5311/a2c-PandaPushDense-v3-pushshaping

A2C Agent playing PandaPushDense-v3 (with reward shaping)

An A2C agent trained on PandaPushDense-v3 from panda-gym using stable-baselines3.

Reward shaping

The default dense reward is -||block_pos - goal_pos||. It does not change until the gripper touches the block, so there is no signal guiding the arm toward it. During training, an extra term rewards moving the gripper toward the block:

reward = original_dense_reward - 0.5 * ||ee_pos - block_pos||

• ee_pos = obs["observation"][0:3], block_pos = obs["observation"][6:9]
• The shaping is applied only during training. Evaluation uses the unmodified environment.

Training setup

Item	Value
Algorithm	A2C (MultiInputPolicy, SB3 default hyperparameters)
Parallel envs	4
Total timesteps	1,000,000
VecNormalize	norm_obs=True, norm_reward=True, clip_obs=10, clip_reward=10

Usage

import gymnasium as gym
import panda_gym
from huggingface_sb3 import load_from_hub
from stable_baselines3 import A2C
from stable_baselines3.common.vec_env import DummyVecEnv, VecNormalize

repo_id = "chrisluo5311/a2c-PandaPushDense-v3-pushshaping"
model = A2C.load(load_from_hub(repo_id, "a2c-PandaPushDense-v3-pushshaping.zip"))

env = DummyVecEnv([lambda: gym.make("PandaPushDense-v3", render_mode="rgb_array")])
env = VecNormalize.load(load_from_hub(repo_id, "vec_normalize.pkl"), env)
env.training = False      # do not update normalization stats
env.norm_reward = False   # report the raw reward

obs = env.reset()
for _ in range(1000):
    action, _ = model.predict(obs, deterministic=True)
    obs, reward, done, info = env.step(action)