YauhenBichel/op3-humanoid-walking-ppo

OP3 humanoid walking policies (PPO, MuJoCo Playground)

Walking policies for the ROBOTIS OP3 humanoid (about 50 cm, 20 joints) in simulation, trained with PPO (Brax) in MuJoCo Playground's Op3Joystick and exported to plain NumPy, so the robot's computer needs no JAX. Simulation only: not validated on hardware.

The baseline walking 10 s at 0.5 m/s in the control loop meant for the real robot.

folder	what	result
baseline/	trained to follow velocity commands	walks 5.27 m in 10 s; 0 % falls in 128 test episodes; usable 0.3–0.8 m/s; falls at any shove from 30 N
staged-shoves/	fine-tuned with growing shoves, servo delay, sensor offsets and random physics	falls in 20.3 % of shoved, randomized episodes (goal: under 10 %); no falls at 15 N; still falls at 30 N; walks 4.67 m in 10 s; usable only 0.4–0.5 m/s

Both are honest intermediate results of an ongoing project, not finished controllers.

Robustness in the robot's control loop

Each run: 6 s at 0.4 m/s in plain MuJoCo with the 50 Hz loop and its safety stops (a fall = tilt past the training limit). Conditions: servo delay of 1–2 control periods, servo stiffness ×0.7/×1.3, 0.5 kg on the torso, sliding friction 0.4, 0.05 rad encoder offsets, IMU noise, and a 0.1 s shove at 2 s. hardened-02 is a first hardening attempt with 1–5 m/s kicks, which did not learn (not published).

Command vx 0.4 m/s for 6.0 s in C MuJoCo, the deployment control loop and its safety stops. Cells: fall rate (runs), mean forward speed m/s.

condition	baseline	hardened-02	staged-shoves
nominal	0 % falls (1), 0.40	0 % falls (1), 0.37	0 % falls (1), 0.37
latency_1	0 % falls (1), 0.38	0 % falls (1), 0.34	0 % falls (1), 0.34
latency_2	0 % falls (1), 0.35	0 % falls (1), 0.30	0 % falls (1), 0.32
kp_0.7	0 % falls (1), 0.24	0 % falls (1), 0.25	0 % falls (1), 0.24
kp_1.3	0 % falls (1), 0.53	0 % falls (1), 0.46	0 % falls (1), 0.48
mass_+0.5kg	0 % falls (1), 0.36	0 % falls (1), 0.34	0 % falls (1), 0.34
friction_0.4	0 % falls (1), 0.37	0 % falls (1), 0.35	0 % falls (1), 0.36
encoder_0.05	0 % falls (8), 0.37	0 % falls (8), 0.40	0 % falls (8), 0.38
imu_noise	0 % falls (8), 0.41	0 % falls (8), 0.37	0 % falls (8), 0.38
push_15N	38 % falls (8), 0.35	0 % falls (8), 0.36	0 % falls (8), 0.37
push_30N	100 % falls (8), 0.34	88 % falls (8), 0.32	100 % falls (8), 0.29
push_60N	100 % falls (8), 0.34	100 % falls (8), 0.33	100 % falls (8), 0.30
push_90N	100 % falls (8), 0.36	100 % falls (8), 0.34	100 % falls (8), 0.31

The same 30 N shove: baseline (left), staged-shoves policy (right). Both fall.

How it was trained

• Environment: MuJoCo Playground 0.2.0 Op3Joystick (OP3 model from MuJoCo Menagerie), control every 0.02 s, observation of 147 numbers (gyro, gravity direction, command, joint angles, last action; 3 frames), 20 actions, joint targets = default pose + 0.3 × action.
• Network: MLP 4 × 128, swish; output 2 × 20 (mean, scale); action = tanh(mean).
• Baseline: Brax PPO, 103 M steps in 96 minutes on a 16-core CPU (AMD Ryzen AI MAX+ 395, JAX split into 16 CPU devices).
• Staged shoves: stage 1 (kicks of 0.1–1.0 m/s every 2–5 s) stopped early when the training ran out of memory; stage 2 continued from its checkpoint for about 69 M steps with kicks of 0.3–2.0 m/s every 1.5–4 s, servo delay in half the episodes, ±0.03 rad encoder offsets and random friction, masses and servo gains.

Use

import numpy as np

z = np.load("policy.npz")
n = int(z["n_layers"])
W, b = [z[f"w{i}"] for i in range(n)], [z[f"b{i}"] for i in range(n)]

def act(obs):
    """Deterministic action in [-1, 1] for one observation (Brax PPO inference, no JAX needed)."""
    x = (obs - z["obs_mean"]) / z["obs_std"]
    for w, bias in zip(W[:-1], b[:-1]):
        x = x @ w + bias
        x = 0.5 * x * (1 + np.tanh(0.5 * x))          # swish
    out = x @ W[-1] + b[-1]
    return np.tanh(out[: int(z["action_size"])])

targets = z["default_pose"] + z["action_scale"] * act(obs)   # joint position targets

Observation layout and joint order follow Playground's Op3Joystick; actuator_names and default_pose are stored in the file. A complete control loop, simulator and talking-robot demo that loads this format: humanoid-companion. All experiments behind these numbers: YauhenBichel/humanoid-lab-experiments.

Limits

Simulation only; the robot falls at shoves of 30 N and more; the staged-shove policy walks usefully in a narrower speed range than the baseline. Not a safety-rated controller: a real robot running it can fall, pinch or hit. Not a medical or care device.

Credits: MuJoCo Playground and MuJoCo Menagerie (Google DeepMind, Apache-2.0). Not affiliated with ROBOTIS or Google DeepMind.