Rice-RobotPI-Lab/robotok-public

RoboTok — An Internet-Scale Data Engine for Human Demonstration Video Retrieval and Dexterous Manipulation Learning

🌐 Project website: rice-robotpi-lab.github.io/RoboTok

Released checkpoints and evaluation keypoints for RoboTok, a model that retrieves web video clips by 3D hand-motion similarity. Similarity is defined by DTW over torso-relative 3D hand keypoints; the encoder is trained to reproduce that DTW ranking in a fast embedding space.

Training and evaluation code is in the accompanying source release.

Files

File	Size	Description
models/best_abs_retrieval_model.pt	11 MB	Retrieval encoder. Cross-attention head over [T, 126] hand-trajectory features (21 joints x 3 coords x 2 hands, T_max = 42): 1 learned query token, 256-d input projection, 1 cross-attention layer (4 heads, sinusoidal PE), 2-layer MLP to a 256-d embedding. DTW design abs_21j_coords.
models/best_abs_retrieval_model.yaml	2 KB	Minimal config to reload the encoder for inference.
models/body_pose_est.pt	9.9 MB	Vector-neuron torso/body-frame estimator: 4-layer rotation-equivariant transformer mapping two-hand trajectories to a torso frame, with separate rotation and translation heads.
eval_data/torso_relative_clip_keypoints.pt	6.5 GB	Torso-relative 3D hand keypoints per clip. Each entry has video_number, node_number, node_uid, keypoints_per_frame (kpts_2d, kpts_3d), and infilled / depth_grounded flags.

Loading

import torch

ckpt = torch.load("models/best_abs_retrieval_model.pt", map_location="cpu", weights_only=True)
ckpt["head_state_dict"]   # encoder weights
ckpt["config"]            # full training configuration

vn = torch.load("models/body_pose_est.pt", map_location="cpu", weights_only=True)
vn["model"]               # torso estimator weights

Citation

@article{qian2026robotok,
  title     = {RoboTok: An Internet-Scale Data Engine for Human
               Demonstration Video Retrieval and Dexterous Manipulation
               Learning},
  author    = {Qian, Howard and Chen, Yiting and Xie, Yunfei and
               Ren, Kejia and Chanrungmaneekul, Podshara and Wang, Gaotian and
               Wen, Bowen and Wei, Chen and Hang, Kaiyu},
  journal   = {arXiv preprint arXiv:2609.03199},
  year      = {2026}
}

License

FAIR Noncommercial Research License v1 (see LICENSE-Action100M). Noncommercial research only.

The released checkpoints (models/*.pt) and evaluation keypoints (eval_data/torso_relative_clip_keypoints.pt) are derivative works of Action100M (Meta FAIR) clips and are governed by that license. It covers trained model weights as "Research Materials", and restricts both those materials and any outputs or results obtained from them to noncommercial research use. If you publish results obtained using these materials, the license requires you to acknowledge that use.

MIT (LICENSE) covers only models/best_abs_retrieval_model.yaml and the accompanying source release.

The MANO / SMPL-H body models required by parts of the pipeline are not included and remain under their own MPI-IS license terms — register at https://mano.is.tue.mpg.de to obtain them.