DeeJanuz/frametop-hands-models

Frametop hand models

Hand-tracking models for the infrared tracking cameras of the Valve Steam Frame headset. They're the two models Frametop's hand tracker (ft-hands) runs: MediaPipe's palm detector, which finds a hand, and its hand landmark model, which places 21 points on it. The network graphs are unchanged; only the weights were retrained, on the headset's own camera images from the Frametop Hands dataset.

Frametop downloads these files when it builds its hand tracking, from a fixed commit of this repository, and checks each file's SHA-256. You don't need to download them yourself.

Frametop is an independent open-source project. It isn't affiliated with or endorsed by Valve, Google or Hugging Face. Steam and Steam Frame are trademarks of Valve Corporation.

Files

Each model version is a folder, so earlier versions stay available at the same paths.

File	What it is
<version>/palm.ncnn.param, palm.ncnn.bin	the palm detector (192 x 192 input), ncnn, fp16
<version>/hand.ncnn.param, hand.ncnn.bin	the hand landmark model (224 x 224 input), ncnn, fp16
<version>/NOTICE	how that version was trained and what it was tested on

The folders hold no 8-bit (*-int8) files, so ft-hands' --int8 option uses its stock models instead.

Why retrain

The original MediaPipe models were trained on colour photos from ordinary lenses. The Steam Frame's tracking cameras are monochrome, infrared and fisheye. On the headset the original models lose hands often, and in bright rooms they miss most of them. Replaying the maintainer's recordings through ft-hands, the retrained pair tracked more than twice as many hands in a bright room as the originals, and 1.3 to 1.8 times as many in dim and lit ones. On held-out contributors (people whose recordings were never trained on), the retrained landmark model's points are about a quarter closer to the labels and its 3D hand shape about 40% closer. Each version's NOTICE has its numbers.

Training

• Data. Contributors' sessions from the Frametop Hands dataset, the maintainer's own recordings, and short clips of tracking failures the maintainer flagged while using the tracker. Sessions with controllers in the hands were left out of bare-hand training.
• Labels. Made automatically: a teacher model (DWPose for the hand's position, then RTMPose for the hand itself, both Apache-2.0) on every camera view, triangulated across the headset's cameras, smoothed, and fitted with a hand skeleton. Where the teacher missed a hand, an earlier retrained landmark model filled it in, through the same triangulation. Nobody labelled images by hand.
• Method. Fine-tuning from the original weights, with the crops cut the way ft-hands cuts them. The training code is in Frametop's lab repository and isn't published yet.

Limits

• Made for the Steam Frame's infrared tracking cameras and Frametop's tracker. On other cameras they'll likely do worse than the originals.
• Trained on a small number of people (under 15) and rooms. Hands, skin, sleeves, jewellery and rooms unlike theirs may track worse.
• The labels come from a teacher model, not from motion capture, so their errors are partly baked in.

Licence

These weights are licensed under Creative Commons Attribution-NonCommercial 4.0 (CC BY-NC 4.0; the full text is in LICENSE), the same licence as the dataset they were trained on. You may use and share them for non-commercial purposes, with attribution to "Frametop (DeeJanuz)". This is not Frametop's MIT licence: the code that runs them is MIT, but these model files are not.

The original MediaPipe models (Google LLC), as converted to ONNX by the OpenCV Zoo, are licensed under the Apache License 2.0. The graphs here are theirs.

You agree not to use these models to identify, recognise, track or profile people, or for biometric identification.