HypnosXC/JoHan_pretrained

JoHan pretrained: ViT-H/512

Paired RAW step 145000 inference weights for JoHan, Video-Conditioned Generative Joint 2D–3D Hand Motion Recovery.

Paper · Project

Contents

File	Purpose
checkpoint/model.safetensors	JoHan joint 2D–3D hand motion network
checkpoint/dino_finetuned.pt	Matching fine-tuned DINOv3 ViT-H/16+ feature extractor
model_config.yaml, run_config.yaml	Plain inference configuration

Both weights come from the same v3 step 145000 checkpoint (training source 2ff60ef). Do not mix them with another checkpoint or EMA mode. This release does not include EMA weights, optimizer state, training data, MANO assets or detector weights.

Download and use

hf download HypnosXC/JoHan_pretrained --local-dir assets/JoHan_pretrained

Install the inference code and follow its asset setup instructions for DINOv3, WiLoR and MANO. From the code checkout:

johan --video input.mp4 \
  --checkpoint assets/JoHan_pretrained/checkpoint --weights raw \
  --dino-repo assets/dinov3 \
  --mano-root assets/mano --detector assets/detector.pt \
  --intrinsics 1000 1000 640 360 --output outputs/predictions.npz

Replace the example intrinsics with calibrated fx, fy, cx, cy in original-video pixels. The loader constructs the DINOv3 architecture without pretrained weights, then strictly loads the complete trained feature-extractor state from checkpoint/dino_finetuned.pt. No separate pretrained initialization is required. The coordinate scale is included in model_config.yaml.

Scope and validation

This model uses 512-pixel RGB images, 128-frame temporal windows, and the original 10-step sine flow schedule. It predicts local MANO pose/shape and 2D joints; the inference pipeline recovers camera-space orientation and translation via PnP. It does not estimate world-space trajectories or body pose. The detector-based pipeline currently targets a single person, handles missing detections as invalid, and keeps decoded video frames in memory.

Validation includes strict loading of these RAW weights, exact agreement with the original sampler on a two-frame synthetic-feature check (four outputs, maximum absolute difference 0.0), and a real-video CPU smoke run using two frames/two flow steps with four detected hand observations. These are integration checks, not a full-video accuracy or GPU performance benchmark. See code validation notes.

Licenses

JoHan weights use MIT. Fine-tuned DINOv3 weights remain subject to Meta's DINOv3 License; see LICENSE.md and the included full DINOv3 agreement. DINOv3, MANO and the WiLoR detector must be credited and used under their respective terms.

Citation

@misc{xu2026johan,
  title={Video-Conditioned Generative Joint 2D-3D Hand Motion Recovery},
  author={Xu, Chen and Li, Yunqi and Huang, Binbin and Yi, Brent and Gao, Shenghua and Ma, Yi},
  year={2026},
  eprint={2610.10512},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2610.10512}
}