mmm

Retrieval augmented state space model for real time character animation: a Mamba backbone that takes a joystick request, its own previous pose, and k poses retrieved from a motion matching database, and predicts the next pose.

This is a Phase 1 checkpoint (pre-training on mocap).

This model is not usable on its own

The retrieved candidate poses are part of the input, not an optional extra. Without the matching database the model is being fed zeros in 371% of its input width and will not produce correct motion.

The database this checkpoint was trained against:

fingerprint	f2afa3c11db090a3
rows (frames)	978,767
pose width	151
frame time	0.016666666666666666 s
clips	77 (aiming1_subject1, aiming1_subject4, aiming2_subject2, aiming2_subject3...)

The database is built from LaFAN1 and is not distributed here. Rebuild it with scripts/build_databases.py; the fingerprint above must match, or the retrieved poses are in a different space from the ones the model learned on.

Architecture

backbone	MambaBackbone
n_layers	4
d_model	256
d_state	16
d_conv	4
d_inner	512
head	full
pose_dim	151
joystick_dim	12
n_candidates	4
input_dim	163
output_dim	153
predict_contacts	True
parameters	1,833,625
use_retrieval	False

Training

epoch	train	validation
1 (first)	1.3656	0.9081
48 (best)	0.2830	0.2620
50 (last)	0.2826	0.2622

Trained for 50 epochs.

Retrieval gain

The number this project is actually judged on. It is the validation loss with the retrieved candidates zeroed, minus the loss with them present, so positive means retrieval is helping.

It is reported in two regimes and only the free running one is meaningful. Under teacher forcing the model is handed the true previous pose every frame, and at 60Hz that is a far better predictor than any retrieved candidate -- measured on this database, copying the previous pose scores 2.29 while copying the nearest candidate scores 16.24. A teacher forced model is therefore right to ignore retrieval, and its gain sits at zero whatever the model has learned.

regime	with retrieval	without	gain
teacher forced	0.2620	0.2620	+0.0000
free running	0.5766	0.5766	+0.0000

Provenance

• Rotation maths cross validated against GenoViewPython's quat.py (MIT, Copyright (c) 2024 Daniel Holden) to 1e-9.
• Spring damper after Holden, Spring-It-On; inertialization after Bollo, GDC 2018; 6D rotations after Zhou et al., CVPR 2019.
• Learned Motion Matching (Holden et al., SIGGRAPH 2020) is background and baseline for this work, not its architecture.

See NOTICE.md in the source repository for the full attribution.

Format

safetensors, with the run's architecture, config and calibrated loss weights JSON encoded in the metadata header. No pickle is involved in loading.

from safetensors import safe_open

with safe_open("published.safetensors", framework="pt") as f:
    print(f.metadata()["architecture"])