chrisluo5311/ppo-3DBall

PPO Agent playing Unity 3DBall (with domain randomization)

A PPO agent trained with the Unity ML-Agents Toolkit on the 3DBall (3D Balance Ball) environment, using domain randomization over the ball's mass and scale.

The agent tilts a cube platform on two axes to keep a ball balanced on top of it for as long as possible — while the ball's physical properties are resampled every episode.

A replay is playable in the Video Preview panel at the top of this page (replay.mp4) — all 12 agents in the 3DBall scene running this policy in Inference Only mode, recorded in the Unity Editor for 26 s at 720×720.

Note that mass and scale show their Unity defaults of 1.0 in that recording. The randomized values are pushed in by the Python trainer over the environment-parameters side channel, so they apply during training only — not during plain Editor playback.

Results

Mean reward	100.00 ± 0.00
ML-Agents benchmark for 3DBall	100
Mean episode length	999 / 999 steps (never drops the ball)
Steps to convergence	~132,000
Total training steps	500,400
Wall-clock training time	5 min 24 s (Apple Silicon, time_scale: 20)

The reward ceiling for this task is 100 (+0.1 per step × 1000 agent decisions per episode), so the agent is fully solved: it holds the ball for the entire episode regardless of how the mass and scale are sampled.

Training curve

Step	Cumulative Reward	Episode Length
12,000	1.01	20.1
48,000	2.78	37.8
72,000	10.81	119.9
84,000	34.38	361.4
96,000	70.38	733.2
108,000	91.42	912.5
120,000	99.66	999.0
132,000	100.00	999.0
492,000	100.00	999.0

Reward reaches the ceiling at ~132k steps. Between 180k–400k it occasionally dips to ~86–95 as the randomization sampler draws harder mass/scale combinations, then stays pinned at 100 from 408k onward.

Final diagnostics (step 492,000)

Metric	Start	End
Losses/Value Loss	0.2008	0.0028
Losses/Policy Loss	0.1009	0.0998
Policy/Entropy	1.4189	1.2338
Policy/Extrinsic Value Estimate	-0.0857	10.0023
Policy/Learning Rate	3.0e-4	~0 (linear decay)
Policy/Epsilon	0.1976	0.1037

Environment

Environment	3DBall (Unity ML-Agents example)
Scene	Assets/ML-Agents/Examples/3DBall/Scenes/3DBall.unity
Parallel agents	12 (identical, shared Behavior Parameters)
Observation space	8 continuous — cube rotation (2), ball relative position (3), ball velocity (3)
Action space	2 continuous — X-rotation, Z-rotation
Reward	+0.1 per step the ball stays on the cube; -1.0 when it falls off
Episode limit	MaxStep: 5000 with DecisionPeriod: 5 → 1000 agent decisions

Domain randomization

Sampled uniformly at the start of every episode (environment_parameters in the config):

Parameter	Sampler	Min	Max	Unity default	Seed
mass	uniform	0.5	10.0	1.0	1427
scale	uniform	0.75	3.0	1.0	1428

This is what makes the run harder than vanilla 3DBall — the agent cannot memorize one ball's dynamics, it has to produce a policy robust across a 20× mass range and a 4× size range.

Training configuration

Trainer: PPO, 500,000 max steps.

Hyperparameters

Parameter	Value
trainer_type	ppo
batch_size	64
buffer_size	12000
learning_rate	3.0e-4
learning_rate_schedule	linear
beta (entropy regularization)	1.0e-3
beta_schedule	linear
epsilon (PPO clip)	0.2
epsilon_schedule	linear
lambd (GAE)	0.99
num_epoch	3
shared_critic	false

Network settings

Parameter	Value
normalize	true
hidden_units	128
num_layers	2
vis_encode_type	simple
memory (LSTM)	none

Reward signals

Parameter	Value
extrinsic.gamma	0.99
extrinsic.strength	1.0

Run settings

Parameter	Value
max_steps	500000
time_horizon	1000
summary_freq	12000
keep_checkpoints	5
checkpoint_interval	500000
threaded	false
num_envs / num_areas	1 / 1
seed	-1 (random)

Engine settings

Parameter	Value
time_scale	20
quality_level	5
capture_frame_rate	60
target_frame_rate	-1
no_graphics	false (trained in the Unity Editor)

Software versions

ml-agents	1.2.0.dev0
ml-agents-envs	1.2.0.dev0
Communicator API	1.5.0
PyTorch	2.8.0
Python	3.10
Unity	6.0 (6000.0.77f1), Apple Silicon / Metal

The complete resolved config is in configuration.yaml (and as JSON in config.json).

Repository contents

File	Description
3DBall.onnx	The trained policy — drop this into Unity
3DBall/checkpoint.pt	Full PyTorch training state, required for --resume
3DBall/events.out.tfevents.*	TensorBoard scalars
configuration.yaml	Full resolved training configuration
config.json	Same config as JSON
replay.mp4	26 s inference replay of all 12 agents; drives the Video Preview widget
run_logs/timers.json	Per-phase timing breakdown
run_logs/training_status.json	Checkpoint registry and final reward

Usage

In the Unity Editor

1. Open the 3DBall scene from the ML-Agents examples.
2. Download 3DBall.onnx and drop it into your Unity project's Assets/.
3. Select the 3DBall agent prefab → Behavior Parameters → assign the .onnx to Model.
4. Set Behavior Type to Inference Only and press Play.

Watch in the browser

1. Go to https://huggingface.co/unity
2. Enter the model id: chrisluo5311/ppo-3DBall
3. Select 3DBall.onnx and click Watch the agent play 👀

Reproduce the training

mlagents-learn config/ppo/3DBall_randomize.yaml --run-id=first3DBall_randomizeRun
# then press Play in the Unity Editor within 60 seconds

Resume from this checkpoint

mlagents-learn config/ppo/3DBall_randomize.yaml --run-id=first3DBall_randomizeRun --resume

References

• Unity ML-Agents Toolkit
• ML-Agents documentation
• Hugging Face Deep RL Course — Unit 5