Sky-Kim/com.sky.fullbody-ik

Analytic Full-Body IK for Unity

Closed-form full-body IK from 6–13 trackers (or MediaPipe Pose landmarks) that drives any humanoid rig — Unity Humanoid, Mixamo, SMPL-X, SMPL and VRM — out of the box. Pure C#, no dependencies, ~9 µs per avatar. This repository is the Unity package itself (package.json at the root).

▶ showcase.mp4 · all formats in sync · per format: Humanoid · Mixamo · SMPL-X · VRM · Generic — captured in Unity play mode (Tools~/capture_demos.sh). 6 vs 13 trackers (red spheres = trackers, same motion frame for frame): showcase · overview · Humanoid · Mixamo · SMPL-X · VRM · Generic.

At a glance

cost per avatar (6 trackers, N = 256)	8.7 µs
error, 6 trackers, captured motion	3.1 cm
error, 11 trackers	1.1 cm
error, 13 trackers (+ shoulders)	0.02 cm
joint rotations for skinning	every joint
rigs	Humanoid, Mixamo, SMPL-X, SMPL, VRM — auto-detected
license	MIT

Mean joint position error (MPJPE) against a captured motion clip; timings on an Apple M5 Max, editor play mode. With six trackers the body between them is estimated; every extra tracker is hit exactly.

What it does

• Spine — FABRIK over Spine → Chest → UpperChest → Neck between the pelvis and HMD (bone lengths kept), warm-started from a blended guess; a chest tracker pins it.
• Limbs — analytic two-bone IK with a body-relative pole; elbow/knee trackers set the bend, shoulder trackers aim the collars.
• Rotations — a global rotation for every joint, built so that forward kinematics of the rotations reproduces the solved positions (aim + twist reference, hinge frames for elbows/knees, tracker rotations for head/hands/feet).
• Rig binding (RigProfile) — finds the 22 joints (Humanoid Avatar, or bone-name tables), measures the body (each rig's own proportions), works out the rig's frame (Z-up export, ±Z facing, cm scale), swings A-pose limbs onto a T-pose, and computes a bind rotation per bone. Missing optional joints (UpperChest, Neck, collars, toes) are handled.
• MediaPipe Pose (MediaPipePose) — 33 landmarks → 6, 12 or 13 trackers (head from ears + nose, hands from wrist/index/pinky, feet from ankle/heel/toe, pelvis from the hip line; shoulders, elbows, knees from their landmarks; the chest estimated from the trunk), body calibration from landmarks, pose_world_landmarks → Unity, and Mirror for selfie video.

Install

Package Manager ▸ + ▸ Add package from git URL…

https://huggingface.co/Sky-Kim/com.sky.fullbody-ik.git

Requires Unity 6 (6000.0+; developed on 6000.3) and git LFS (the sample models and the documentation videos are LFS files). Pin a version with #<commit> at the end of the URL. Then Samples ▸ Format Demos ▸ Import for the demo scenes, and MediaPipe Video Demo for the video → pose → IK demo (needs Sentis).

Quick start

1. Put your model in the scene (Humanoid, Mixamo FBX, SMPL-X, VRM …) in its rest pose.
2. Add Component ▸ Analytic Full-Body IK ▸ Full Body IK. The inspector detects the format and shows the bone map (pick a bone to override; yellow = optional joint the rig lacks) and the measured body.
3. Click 6-point or 13-point, then Create Tracker Objects — targets appear at the bones.
4. Parent each target under its tracked device (HMD, controllers, Vive trackers …) keeping the offset. Done: it solves every LateUpdate, after the Animator, and Weight blends it over animation.

Without hardware, add Motion Tracker Driver and give it a motion JSON (the sample's medley.json).

// your own tracker source (e.g. MediaPipe): build frames in code — Tracker Input = Code, no tracker objects
var set = TrackerSetSpec.MediaPipe("13pt", "Spine3", "L_Shoulder", "R_Shoulder", "L_Elbow", "R_Elbow", "L_Knee", "R_Knee").Resolve();
ik.Initialize(set);                                       // also fine at runtime: the rig is reset to rest first
for (int i = 0; i < 33; i++) landmarks[i] = MediaPipePose.FromMediaPipeWorld(worldLandmarks[i]);
if (selfie) MediaPipePose.Mirror(landmarks);              // x flipped and left/right swapped
for (int i = 0; i < 33; i++) landmarks[i] += playSpaceOffset;
ik.SolveFrame(MediaPipePose.ToTrackers(landmarks, ik.Body, set));

// or a crowd without components: one solver, N avatars, mixed bodies
var solver = new AnalyticFullBodySolver();
solver.Initialize(n, TrackerSet.Base6, bodies);          // SkeletonDef per avatar (RigProfile.Skeleton)
solver.Solve(frames, n, positions);
solver.GetJointRotations(n, rotations);                  // then RigProfile.ApplyPose per avatar

Rig formats

format	detected by	quirks handled
Unity Humanoid	Animator with a humanoid Avatar	any bone names, A-pose, arbitrary bone axes, no UpperChest/Neck, helper bones
Mixamo	mixamorig:Hips … (any prefix:)	Armature ×0.01 (cm), bone +Y axis
SMPL-X	pelvis, left_hip, spine1 …	Z-up authoring, root node, jaw/eye/finger joints ignored
SMPL	m_avg_Pelvis / f_avg_*, Pelvis …	SMPL Unity package naming
VRM	VRoid J_Bip_C_Hips …; other VRMs via the Humanoid Avatar UniVRM creates	−Z facing, anime proportions

Trackers: Head, Left/Right Wrist, Pelvis, Left/Right Ankle (required) + optional UpperChest, Left/Right Shoulder, Elbow, Knee. Bone Space rotations (default) = the world rotation the bone should have.

MediaPipe Pose

tracker points	from the landmarks	mean joint error, synthetic / mocap*
6	head, wrists, pelvis, ankles (+ their rotations from rigid landmark groups)	—
12	+ shoulders, elbows, knees	0.24 / 0.27 cm
13	+ chest: shoulder line + an "up" halfway between hips→shoulders and shoulders→ears	0.13 / 0.19 cm

* clean landmarks generated from the pose (MediaPipePose.FromJoints): the procedural test motion, and a Mixamo motion-capture medley (local only, not in the repository). MediaPipe has no landmark for the spine, neck, collars or toes, and the solver reads at most 13 trackers — every mode still rotates all 22 bones.

Selfie video: use MediaPipePose.Mirror(landmarks). Negating x alone leaves the left landmarks on the body's right — a reflected body the solver cannot follow (28 cm mean error vs 0.26 cm with Mirror).

Motions

The samples ship a procedural motion library (no motion-capture data, regenerate with python3 Tools~/motion_library.py): walk_turn, dance, boxing, jumping_jacks, kicks_lunges, pickup_reach, kneel_wave, and medley (all of them in sequence, 54 s, with named sections — the demo HUD shows the current action). Mean joint error per action against the ground truth (clean trackers):

trackers	walk/turn	dance	boxing	jumping jacks	kicks/lunges	pick-up/reach	kneel/wave
6	1.1 cm	1.5 cm	2.9 cm	0.6 cm	0.6 cm	0.7 cm	0.6 cm
13	0.0	0.1	0.1	0.0	0.0	0.1	0.0

Real motion capture works too: put animation FBX (e.g. Mixamo, any rig the package binds) in Assets/ThirdParty/Animations/, run Analytic Full-Body IK ▸ Dev ▸ Convert Animations (ThirdParty) To Motions (each clip is sampled on its own rig and read back through RigProfile), and python3 Tools~/motion_library.py --mocap Assets/ThirdParty/Motions to chain them into mocap_medley.json; the real-model scenes use it automatically. Converted data stays in the git-ignored folder.

Samples

MediaPipe Video Demo — a video on the left, MediaPipe BlazePose (on Sentis) landmarks over it with the trackers in use marked, and procedural characters on the right following the person through MediaPipePose.ToTrackers → FullBodyIK.SolveFrame. Swap the video in the inspector (also during Play), pick 6 / 12 / 13 tracker points, mirror selfie video, add your own characters. Needs com.unity.ai.inference 2.2+ (the sample's assemblies are skipped without it); Built-in pipeline and URP. No video ships with it — assign your own clip. Models: unity/inference-engine-blaze-pose (Apache-2.0).

Format Demos — Overview (all five side by side) and one scene per format: a procedural mannequin skinned to that format's rig, driven live by the motion medley through six virtual trackers (red spheres) (toggle Extra Trackers on Format Demo for 13), the ground-truth skeleton beside it, and a HUD with the live error and solve time. Drop your own model into Replacement Model on Format Demo to run the same scene on it.

Try it on real models (Mixamo, VRM)

The development tools in Tools~/Dev (copy them into a project's Assets/Dev) build a demo scene for any model you drop in — same trackers, motion, ghost and HUD as the format demos. Models are licensed by their authors, so you download them yourself; Assets/ThirdParty/ is git-ignored and nothing from it is ever committed.

1. Get the models
   • Mixamo — mixamo.com (Adobe account) → pick a character → Download: Format FBX for Unity, Pose T-pose. Copy the .fbx to Assets/ThirdParty/Models/.
   • VRM — any .vrm you may use, e.g. the VRM Consortium sample Seed-san.vrm (VRM Public License 1.0) or a model from VRoid Hub that allows it. Copy it to Assets/ThirdParty/Models/.
2. UniVRM (for .vrm only): add it from Package Manager ▸ + ▸ Add package from git URL (com.vrmc.gltf + com.vrmc.vrm v0.131.3, VRM 1.0 importer): https://github.com/vrm-c/UniVRM.git?path=/Packages/UniGLTF#v0.131.3 and https://github.com/vrm-c/UniVRM.git?path=/Packages/VRM10#v0.131.3. The IK package itself does not depend on UniVRM — it binds the imported VRM through its Humanoid Avatar.
3. Build the scenes: menu Analytic Full-Body IK ▸ Dev ▸ Build Real Model Scenes. Each model gets Assets/ThirdParty/Scenes/Real_<name>.unity; the inspector of its Full Body IK shows what was detected. Mixamo textures are extracted next to the model. To record them like the demos: unity test . --mode PlayMode --filter AnalyticFullBodyIK.Dev.CaptureDemos.CaptureRealModels (frames in Captures/).

Checked so far — all bound with no manual setup, rig error ≤ 1 cm with six trackers:

model	file	detected as	joints
Amy (Mixamo)	FBX, Animation Type Humanoid	Unity Humanoid	22/22
Swat (Mixamo)	FBX, Animation Type Generic	Mixamo (mixamorig: names)	22/22
Seed-san	VRM 1.0 via UniVRM	Unity Humanoid	21/22 (no UpperChest → virtual)

VRM spring bones and constraints (hair, Seed-san's backpack arm) keep running on top of the IK pose. If a model is not in its bind/T-pose when the scene opens, or a bone is detected wrongly, fix it in the Bone Mapping foldout of the inspector.

Repository

package.json, Runtime/, Editor/, Tests/     the package
Samples~/FormatDemos, Samples~/MediaPipeVideoDemo
Documentation~/media/                       the videos above
Tools~/motion_library.py                    regenerates the sample motions
Tools~/Dev/, Tools~/capture_demos.sh        scene builders, capture tests and the recording script; copy Dev into a
                                            development project's Assets/ (Unity ignores folders ending in ~)

Tests (add "testables": ["com.sky.fullbody-ik"] to a project's manifest): unity test <project> --mode EditMode (68) and --mode PlayMode (1).

Notes

• The 22-joint skeleton is SMPL-topology with generic anthropometric proportions — not the SMPL body model.
• The sample motion is procedural (walk, squat, reach, turn), so the repo carries no motion-capture data.
• Rigs without an UpperChest or Neck bone (many Unity Humanoids) bend the spine only at their real bones (AnalyticFullBodySolver.SetRigidJoints, set by FullBodyIK), so head and hands stay on their trackers.
• With six trackers nothing observes how the spine bends relative to the pelvis or which way the elbows point; the solver guesses (blend + body-relative poles). Add trackers where it matters.

License

MIT — see LICENSE.md. The MediaPipe sample's BlazePose models are Apache-2.0 (see the sample README).