vmr3D — Volumetric Motion Rigging
Author and contact: redazul
Experimental assistance: GPT-6-Astra assisted during selected experimental runs. Research direction, final result selection, and Unity validation were provided by redazul.
VMR is a practical workflow for recovering character motion from reconstructed mesh sequences, rebuilding difficult surface detail, and preserving the quality of the final animation.
Release status: research artifacts and preparation for a learned model. This repository currently contains an animated character, demonstrations, validation records, and a proposed learning roadmap. It does not yet contain trained VMR neural-network weights, an implemented learning architecture, or a general video-to-character inference endpoint. The FBX is an animation asset, not a neural-network checkpoint.
What VMR does
Reconstructing a video into independent 3D meshes can reproduce the movement while allowing the character's proportions, grip and surface detail to change between frames. VMR addresses that problem by separating a coarse motion representation from detailed surface reconstruction, then protecting accepted geometry during refinement.
The demonstrated input is a video together with reconstructed reference meshes. The result is a textured character animation that can be opened and inspected in Blender and imported into Unity as an animated FBX. The reference meshes are reconstructed estimates, rather than perfect ground truth.
| Current result | Detail |
|---|---|
| Demonstration | A 52-pose stylized baseball swing |
| Playback | 30 fps, one stored pose per reference frame |
| Export | FBX with animated blend shapes and embedded textures |
| Unity | Author confirmed that the FBX works and its animation is available |
| Export fidelity | All 52 poses checked after fresh FBX reimport; maximum world-space vertex difference below 4.85 × 10⁻⁸ scene units |
| Motion quality | VMR was qualitatively preferred for lower visible jitter among the compared outputs |
| Current limitation | One surface over poses 1–51; a separately preserved closing surface at pose 52 |
How the workflow works
- Establish the reference character. Align the reconstructed targets in a common coordinate system and select detailed reference geometry. Treat camera/alignment differences separately from actual character motion.
- Recover motion with interior volumes. Partition the reference interior into consistent regions and use interior-distance influence weights to guide the surface. In this implementation, 64 volume controls carry rotation, translation and regularized scale, with four influences per original vertex. These controls are not the same as the skeletal approximation tested later.
- Fit the surface while preserving local shape. Correct geometric mismatch while protecting accepted regions. A lower nearest-surface error can still damage a palm, collapse a wrist or pull one contacting surface toward another, so distance alone is insufficient.
- Reconstruct difficult detail explicitly. Recover the hand/grip/bat detail from reference geometry where deformation cannot preserve it. Transport the opening grip coherently through the motion and retain the approved closing configuration.
- Preserve volume and endpoints. Repair local hand/forearm geometry with thickness constraints. Hold the opening and closing checkpoints fixed and validate them with consistent camera views.
- Export and check playback. Preserve the reviewed poses in Blender and FBX, compare evaluated geometry after export, and inspect the resulting animation in the target application.
The contribution is an implemented combination of established techniques with explicit reconstruction and validation steps. VMR is a descriptive name for that workflow; scientific novelty or general superiority is not claimed.
What the comparisons established
Two Trimesh non-rigid registration refinements—Amberg and Sumner/Popovic—were evaluated from the existing animation. Both reduced the measured one-way distance to sampled target geometry. The original VMR result was still preferred visually for its lower apparent jitter. The distance measurement uses samples involved in fitting; it is neither a held-out accuracy test nor a temporal jitter score.
A separate Dem Bones comparison approximated a corresponding 51-pose surface using 64 bones and up to four influences per vertex. It demonstrated an actual skeletal approximation, but it did not resolve the separate closing surface. The downloadable VMR character retains the approved baked motion rather than substituting this approximation.
The evidence is specific to this sequence and the tested configurations. Preserving coherent motion can be preferable to following small inconsistencies in each reconstructed target, but the individual causes of the visual differences have not been isolated.
See the result
Watch the synchronized source / reference mesh / VMR comparison
Input video · Final VMR animation · Actual Blender workspace animation
The Blender demonstration uses real interface captures at each of the 52 stored poses, replayed at 30 fps. It shows editable scene content and timeline poses, not a real-time performance benchmark.
Downloads and the full paper
- Download animated FBX
- Download Blender scene
- Read the complete white paper here on Hugging Face
- Read the interactive white paper on DiscoSea
- Read the PDF on Hugging Face
- Download standalone HTML
- FBX validation record
Both locations contain the full paper. The website adds synchronized video controls; the Hugging Face edition includes figures and direct links to repository-hosted videos.
Use the release
Download the FBX directly for your 3D application. To inspect the complete release locally:
pip install huggingface_hub
hf download DiscoSea/vmr3D --local-dir vmr3D
cd vmr3D
python scripts/verify_release.py
python -m http.server 8080 --directory whitepaper
Open http://localhost:8080/ for the standalone paper. Its media links use Hugging Face and require an internet connection; the corresponding media files are also present in whitepaper/assets/. This serves an existing demonstration; it does not run new reconstruction. The current release provides no automatic Hugging Face inference widget.
For an Animator Controller in Unity, import the FBX as Generic animation, enable blend shapes and animation, and disable animation compression. Add the imported clip to a controller on the model root. The final-surface handoff can be interpolated between sampled poses; no Unity helper package is required or included. Blender playback is available through the supplied scene.
Preparing the learned model
The intended next capability is video → consistent character representation → motion and local corrections → editable animation. The first proposed training task is deliberately bounded: predict temporally consistent VMR controls and surface corrections from a reference character and a video segment. The existing fitting workflow can act as a teacher and refine early predictions.
The intermediate artifacts have several possible uses:
| Artifact | Potential supervision |
|---|---|
| Video frames and corresponding reconstructed meshes | Appearance-to-geometry relationships, with uncertainty recorded |
| Motion controls and fitted poses | Consistent deformation over time |
| Geometry before and after repair | Local correction targets |
| Protected endpoints and contact regions | Constraints that preserve important details |
| Competing outputs and quality judgments | Preference examples once enough varied comparisons are collected |
| Validated exports | Required output structure, timing and geometry checks |
Many meshes from one sequence do not provide the diversity of many independent characters and motions. Rejected outputs need failure labels; they should not all be treated as correct targets. The final-surface handoff must be represented explicitly until continuous correspondence is solved.
A training program would combine synthetic videos from known animated assets with reviewed reconstructions of varied videos. Entire identities and source motions should be held out for evaluation, keeping neighboring frames and all variants of the same source in one split. Dense predicted motion could later be reduced to editable keyframes within a documented error tolerance.
See the learning roadmap and proposed data contract. These are plans; no VMR neural training run, trained weights or generalization results are reported in this release.
Reproducibility and scope
release_manifest.json records SHA-256 hashes and sizes for the release files. The verification script checks file integrity, FBX format signature, linked local paper assets, and the example record. It does not rerun geometric fitting or certify a new model's quality. The original reconstruction scripts and their complete dependency environment are not packaged in this initial release.
Training data: no neural training run is reported in this release. Evaluation: one 52-pose stylized baseball swing, geometric checks, rendered comparisons, qualitative motion review and author-confirmed Unity use. Failure modes include ambiguous contact, stiff rigidly transported detail, imperfect wrist attachments, and the separate final surface.
Related work and license
The workflow builds on established deformation, registration and skinning ideas. Related implementations include TRELLIS.2, Trimesh and Dem Bones. The paper cites the underlying methods. AniGen is a candidate for future comparison, not an evaluated dependency of this release.
The repository retains its author's Apache-2.0 license designation. Third-party methods, models and source assets retain their applicable terms; this repository does not relicense them. See PROVENANCE.md.