KaptainKris/ppo-LunarLander-v3-flip

PPO LunarLander flip, recover and land agent

This repository contains a Stable-Baselines3 PPO actor-critic agent trained on a customised LunarLander-v3 environment.

Learned task

The curriculum teaches one policy to:

1. complete a full rotation in a fixed direction;
2. recover upright and arrest angular motion;
3. enter the landing zone;
4. reduce descent speed and land safely.

Reward configuration version: v4-soft-touchdown.

Changes in this upload

1. Continued from the selected Phase C checkpoint.
2. Added an altitude-dependent vertical-speed target.
3. Added a near-ground quadratic descent-overspeed penalty.
4. Added a dedicated in-zone crash penalty.
5. Reduced risky late horizontal corrections using a deadband.

Reward design

The shaped reward includes:

• one-off rotation-progress and flip-completion rewards;
• an upright post-flip recovery reward;
• horizontal guidance towards the landing zone;
• an altitude-dependent vertical-speed target;
• attitude and angular-speed control;
• a near-ground descent-overspeed penalty;
• distinct penalties for off-zone landings and in-zone crashes.

Parameter	Value
required_rotations	1
rotation_direction	1
upright_tolerance_radians	0.3
recovery_angular_velocity_tolerance	0.5
pre_flip_original_reward_weight	0.15
post_flip_original_reward_weight	3
rotation_progress_bonus	300
flip_completion_bonus	500
recovery_bonus	250
flip_landing_bonus	2000
landing_without_flip_penalty	300
no_flip_terminal_penalty	300
failed_landing_penalty	800
outside_zone_landing_penalty	500
in_zone_crash_penalty	1200
post_flip_shaping_weight	2
post_flip_shaping_gamma	0.999
post_flip_shaping_clip	30
post_flip_center_weight	80
post_flip_horizontal_speed_weight	50
post_flip_vertical_speed_weight	100
post_flip_angle_weight	70
post_flip_angular_speed_weight	40
post_flip_leg_contact_weight	30
landing_zone_half_width	0.2
post_flip_zone_excess_weight	120
post_flip_target_vx_gain	0.5
post_flip_max_target_vx	0.35
post_flip_horizontal_deadband	0.08
post_flip_target_vy_high	-0.45
post_flip_target_vy_near_ground	-0.12
near_ground_height	0.6
safe_touchdown_vertical_speed	0.18
near_ground_overspeed_weight	120

Evaluation

Deterministic evaluation over 100 fixed-seed episodes:

Metric	Value
Mean shaped reward	-325.25
Mean original reward	-706.46
Full-rotation rate	36.0%
Recovery rate	12.0%
Recovery given a flip	33.3%
Safe-landing rate	0.0%
Flip-and-land rate	0.0%
Terminal in-zone rate	20.0%
In-zone crash rate	8.0%

Architecture

• Algorithm: PPO
• Policy: MLP actor-critic
• Actor hidden layers: [128, 128]
• Critic hidden layers: [128, 128]
• Observation dimensions: 11
• Discrete actions: 4

Training configuration

Parameter	Value
Phase timesteps	2000000
Parallel environments	16
Learning rate	5e-05
Rollout steps per environment	1024
Batch size	64
Optimisation epochs	4
Gamma	0.999
GAE lambda	0.98
Entropy coefficient	0.005
PPO clip range	0.1
Training seed	43

Replay

• Seed: 20016
• Original reward: -679.11
• Shaped reward: -88.08
• Rotations completed: 1.03
• Flip completed: True
• Recovery completed: True
• Landed safely: False
• Outcome: flip_but_failed_landing

Open the full MP4 replay

Repository files

• ppo-LunarLander-v3-flip-128x128.zip: selected PPO model
• flip_landing_reward_wrapper.py: custom environment wrapper
• training_config.json: PPO training settings
• reward_config.json: reward configuration
• episode_results.csv: fixed-seed evaluation episodes
• results.json: machine-readable evaluation summary
• config.json: compact model metadata
• replay.gif: model-card preview
• replay.mp4: full replay