PPO agent for ALE/SpaceInvaders-v5 (from-scratch PyTorch implementation)
This model was trained as part of a project building policy-gradient methods up from first principles: REINFORCE -> A2C -> A2C+GAE -> PPO (this checkpoint).
Environment
ALE/SpaceInvaders-v5, preprocessed with grayscale + 84x84 resize + 4-frame skip + 4-frame stack + terminal-on-life-loss (standard DQN-style preprocessing).
Architecture
Nature CNN trunk (3 conv layers -> 512-d FC) with two heads: policy logits (6 actions) and a scalar state-value.
Training
- Algorithm: PPO, clipped surrogate objective, GAE(lambda=0.95), gamma=0.99
- Rollout length: 128 steps, 4 epochs/rollout, 4 minibatches, clip_eps=0.2
- Optimizer: Adam, lr=2.5e-4
- Total updates: 1000
Evaluation
Mean return over 10 greedy evaluation episodes: 105.0 (std 0.0)
Usage
import torch
from model_def import ActorCritic # see training notebook for the class definition
net = ActorCritic(n_actions=6)
net.load_state_dict(torch.load("ppo_spaceinvaders.pt"))
net.eval()