CLUTCH: Contextualized Language model for Unlocking Text-Conditioned Hand motion modelling in the wild
Authors: Balamurugan Thambiraja
Omid Taheri
Radek Danecek
Giorgio Becherini
Gerard Pons-Moll
Justus Thies
Venue: ICLR 2026 Poster (2026)
Keywords: Human Motion Synthesis
Hand motion synthesis
LLM
Motion in-the-wild
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: CLUTCH is an LLM-based model designed to synthesize and caption natural, in-the-wild 3D hand motions.

Abstract: Hands play a central role in daily life, yet modeling natural hand motions remains
underexplored. Existing methods that tackle text-to-hand-motion generation or
hand animation captioning rely on studio-captured datasets with limited actions
and contexts, making them costly to scale to “in-the-wild” settings. Further,
contemporary models and their training schemes struggle to capture animation
fidelity with text–motion alignment. To address this, we (1) introduce ‘3D Hands
in the Wild’ (3D-HIW), a dataset of 32K 3D hand-motion sequences and aligned
text, and (2) propose CLUTCH, an LLM-based hand animation system with two
critical innovations: (a) SHIFT, a novel VQ-VAE architecture to tokenize hand
motion, and (b) a geometric refinement stage to finetune the LLM. To build 3D-
HIW, we propose a data annotation pipeline that combines vision–language models
(VLMs) and state-of-the-art 3D hand trackers, and apply it to a large corpus of
egocentric action videos covering a wide range of scenarios. To fully capture
motion in-the-wild, CLUTCH employs SHIFT, a part–modality decomposed VQ-
VAE, which improves generalization and reconstruction fidelity. Finally, to improve
animation quality, we introduce a geometric refinement stage, where CLUTCH is
co-supervised with a reconstruction loss applied directly to decoded hand motion
parameters. Experiments demonstrate state-of-the-art performance on text-to-
motion and motion-to-text tasks, establishing the first benchmark for scalable
in-the-wild hand motion modelling. Code, data and models will be released.

Published as a conference paper at ICLR 2026
CLUTCH: CONTEXTUALIZED LANGUAGE MODEL FOR UNLOCKING TEXT-CONDITIONED HAND MOTION MOD- ELLING IN THE WILD
Balamurugan Thambiraja- Omid Taheri', Radek Danecek², Giorgio Becherini",
Gerard Pons-Moll & Justus Thies Technical University of Darmstadt, Germany
Max-Planck Institute for Intelligent Systems, Tuebingen, Germany²
University of Tuebingen, Germany
Tubingen Al Center, Germany"
Max Planck Institute for Informatics, Saarland Informatics Campus, Germany
https: /bba amu rugatthaahmmll www.gai.gtttte. github. i CLUTCH
ABSTRACT
Hands play a central role in daily life, yet modeling natural hand motions remains underexplored. Existing methods that tackle texi-to-hand-motion generation hand animation captioning rely on audio--aptuued datasets with limited actions and contexts, making them costly scale to "in-the- ai-dhh-wiild" settings. Further, contemporary models and their training schemes struggle to capture animation fidelity with text-motion alignment. To address this, we (1) introduce '3D Hands in the Wild' (3D-HIW), dataset of 32K 3D hand-motion sequences and aligned text, and (2) propose CLUTCH, an LLM-based hand animation system with two critical innovations: (a) SHIFT, novel VQ-VAE architecture to tokenize hand motion, and (b) a geometric refinement stage to finetune the LLM. To build 3D- HIW, propose data annotation pipeline that combines vision- language models (VLMs) and state-of-the-art 3D hand trackers, and apply it to a large corpus of egocentric action videos covering wide range of scenarios. To fully capture motion in-the-wild, CLUTCH employs SHIFT. part-modality decomposed VQ- VAE, which improves generalization and reconstruction fidelity. Finally, improve animation quality, we introduce geometric refinement stage, where CLUTCH is co-supervised with a reconstruction loss applied directly to decoded hand motion parameters. Experiments demonstrate state-of- of-the-art performance on text-to- motion and motion-to-text tasks, establishing the first benchmark for scalable in-the-wild hand motion modelling. Code, data and models will be released. Project page:
INTRODUCTION
Hands are at the heart of our daily experiences: With them we write, knit, play instruments, and perform countless other actions that feel effortless to us but remain challenging for generative models reproduce naturally. Capturing this variability is not only essential for natural motion generation, but also foundational for future behavioral AI, where models must infer, predict, and generate human behavior interactive settings such as AR/VR, robotics, and human- computeer collaboration. While prior work has focused on full-body motion, gestures, and hand-object interactions (Chen et al. [2024] Jiang et al. [2024] Liu et al. 2024 Ng et al. 2024 Huang et al., 2025 Christen et al. 2024 Cha et al. 2024; Petrov P et al. 2025), text-guided hand motion generation "in-the-will wild" remains underexplored, with text-to-hand- object interaction methods being the most related line work.
[024|Chha
Hand motion models (Huang al. 2025 Zhou al. 2024 Cha et al. 2024 Zhang al. 2025b)
are primarily trained on high-quality 3D hand motion datasets, such as GRAB Taheri et al. 2020). ARCTIC (Fan et al.) 2023), and H2O (K et al., 2021), all captured in motion capture studios.
However, collecting such datasets is both time-consune consuminn and expensive, limiting scalability to
diverse scenarios and actions. As result, current methods restricted to narrow set of actions

---

Published as a conference paper at ICLR 2026
m
The penson is interacting with laptop"
person playing piano"
"a person kneading flour"
The person is using the hammtrr"
CLUTCH (Tot--o--oCia))
CLUTCH
Figure I: CLUTCH is novel LLM-based model that enables text-conditioned synthesis (left) and
captioning of in-the-wild 3D hand motions (right).
and intents, and cannot generate "in-the-wild" motions. To mitigate this data limitation, we draw inspiration from prior work (Wang et al., 2025: Sklyarova et al. 2023), which leverage VLMs/LLMs as data annotators. Specifically, we integrate 3D hand tracker (Zhang et al., 2025a) with VLM (Wu) et al. 2024) to construct an "in-the-wild" hand motion dataset comprising 32K sequences; approx- imately 10x larger than GRAB and ARCTIC, and larger than the recent Gigahands (Fu et al.) 2025) dataset. We refer to this dataset as '3D Hands in the Wild' (3D-HIW) dataset, which includes multi-action clips like piano and food prep, underrepresented in previous work.
While VLMs demonstrate strong visual understanding, they often hallucinate spurious objects, actions, or concepts (Wu et al. 2025) when captioning. To address this, we introduce a Parallelized Chain-of-Thouge Prompting strategy, which decomposes complex reasoning prompt into multiple atomic prompts, each targeting a specific video aspect. The atomic responses are processed by a summarization module generate an initial annotation, then refined into a more detailed annotation.
Compared to most existing hand motion datasets, which mostly contain single actions or interactions per sequence, in-the-wild hand movements more natural and diverse, often involving multiple actions within the same sequence. This requires motion model that can robustly align hand motion with language representations. Recent approaches, HOIGPT (Huang et al. 2025) and MotionGPT (Huang et al. 2025), repurpose pre-trained LLMs for motion tasks. However, we find that applying them as-is hand animation leads to suboptimal performance due (1) poor generalization capability of the motion tokenizer, and (2) geometric inacurracies in the L.-M-predicted motion. We address this by introducing CLUTCH (Contextualized Language model for Unlocking Text-Conditioned Hand motion). novel LLM for synthesizing and captioning in-the-wild 3D hand motions (illustrated in Fig. D.. In CLUTCH, we address the aforementioned limitations by: (1) novel hand motion prior and (2) new LLM finetuning stage with a geometric refinement loss.
(1) Motion prior. Hand motions are inherently multi-modal. Using standard single VQ-VAE for both hands leads to poor quality of hand motion reconstruction (jitter lack of realism). The diversity of hand motions observed "in-the-wild" exposes this issue further. To address this, we introduce SHIFT (Structuring Hands Into Fine-grained Tokens). SHIFT models trajectory and pose components using separate VQ-VAE's, while disentangling left and right hands during encoding and decoding. Empirically, this formulation achieves stronger generalization and more accurate reconstructions, even under high temporal compression compared to a standard VQ-VAE. It also improves bimanual coordination and reduces jitter.
(2) LLM finetuning. We find that finetuning the LLM on the standard next-token prediction task with the cross-entropy (CE) loss leads suboptimal animation fidelity. We find that token-level accuracy does not guarantee high-quality motion synthesis (as shown in (Hong et al. 2024)). An additional reconstruction loss in motion space needed to improve the motion generation. In CLUTCH, we add a novel geometry refinement stage that decodes the sampled tokens into the hand motion parameters and applies a reconstruction loss directly to the decoded hand motion parameters. This guides the LLM toward selecting codes with stronger animation fidelity. With these, CLUTCH achieves state-of-ft toate-of-the-aad the-art art on wild hand motion synthesis and captioning, and goes beyond studio captures, by generating everyday in-the-wild motions rarely seen in mocap: playing piano (bimanual), cooking, writing, knitting. and more. We show quantitatively, that CLUTCH outperforms recent state-of-the--e methods such as Human MotionGPT, and T2M-GPT.
2

---

Published as a conference paper at ICLR 2026
3. "Hands in the wild' (3D-HIW) dataset generation
4.1. SHIFT Tokenizer
4.2 LLM for hand motion modelling
Conerrte mutioon of Captiaw the givkn mutiow person plaging piana
Reconntracted oction
BP www.ggggrppgggg...
VLM based toxt captioning
3D hand rocoustraction
(Stucturing Hards Fire Guased Tokas)
SHIPY Tohentur
Language Model
Generated captino:
captiie
playing plasso using hands
The jerson vide hamanerr to drice Aolle.
Recoesmarved
Motion bikees
Prsdicned modion
sokree
Figure 2: Overview: CLUTCH is an LLM for synthesizing and captioning in-the-wild 3D hand motions. To train this model, we (i) generate an in-the-wild hand motion dataset (Section 3). We (ii) tokenize the hand motion using a novel decomposed VQ- VAE tokenizer (Section 4.1). We (iii) train the LLM to model both text and motion in unified token space (Section 4.2
The overview of our work is presented in Figure 2 Taken together, our main contributions are: (1) A data acquísition pipeline that combines 3D hand tracker with novel annotation framework driven by a vision-language model to enable scalable in-the-wild 3D hand motion data curation.
(2) Using this pipeline, we construct '3D Hands in the Wild' (3D-HIW), a large-scale dataset comprising over 32K hand motion sequences captured in diverse real-world egocentric videos.
(3) We introduce SHIFT (Structuring Hands Into Fine-grained Tokens) tokenizer, for modelling in-the-wild wild hand motions. SHIFT improves performance over tokenizers used in prior works.
(4) Finally, we propose CLUTCH, an LLM-bbsed generative model for text-conditioned synthesis and captioning of wn-the-wild 3D hand motions; setting new benchmark for scalable in-the-wild wild hand motion modelling.
2 RELATED WORK
Motion Datasets Annotation: Existing motion datasets provide foundation for current human modelling methods. AMASS (Mahmood et al.) (2019) unifies diverse mocap datasets into a large-scale human body motion dataset. While GRAB, ARCTIC, H2O, Dex (Chao al., (021), and OakInk (Zhan et al., 2024; Yang et al. 2022) offer detailed 3D hand-object interactions. More recently, Gigahands (Fu al. | [2255] introduced dataset of 15K hand motion sequences with diverse actions and objects. While these datasets are of high quality, they are costly to collect, confined to controlled studio settings, and cover only narrow action ranges. In contrast, large-scale egocentric datasets such as Ego4D (Grauman et al. 2022) and Ego Vid5M (Wang et al. 2024) capture diverse real-world activities but lack accurate 3D hand reconstructions and textual action deseriptions. Parallel efforts in egocentric video captioning, such as LaViLa (Zhao et al., (023), HOD (Pei al. 2025). and Eg (Hong et al. 2024), leverage language models generating faithful action descriptions from input videos. Li and EgoLM employ large language models (LLMs) to generate dense narrations, while HOD augments narrations (if present) with detected hand oond-ojject trajectories to produce semantically richer descriptions. To enable in the wild hand motion modelling, we construct a large-scale 3D hand motion dataset called '3D Hands in the Wild' (3D-HIW)) based on Ego4D. To this end, we introduce a two-stage annotation pipeline that first applies open-vocabulary reasoning via parallel chain-of-thought prompting, and then refines results with closed vocabl-vocabulary grounding.
Motion Modelling: Research in motion generation has largely focused full-body and gesture
synthesis (Guo et al.
2024]] Liu et al., 2023) Zhang et al. 2023:Diang et al. 2025 Wang et al. 2023)
Shafir et al.
Xie Xie
2023
2023 Karunratanakul et al. |203]
al. 2025c
2023 Athanasiou
et al. 2024 Chi et 20024 Liu et al. 2204). Parallel works have focused on hand object interaction
modelling (Christen et al. [202]] Cha et al.] 2024 Ghosh et al. 2023 Zhou et al. 2022, 2024), built
on MoCap datasets like GRAB ( (Taeri et al. (020) or ARCTIC (Fan et al. 2023). Recent works such
as (Huang et al. 20255 Jiang et al.) 20244] Chen et al. 2024 Li al. 2025a) treat motion tokens as text-like symbols, enabling pretrained LLMs to synthesize motions. While promising, these methods are limited by small-scale datasets and training objectives that emphasize token prediction accuracy rather than reconstruction fidelity. EgoLM (Hong et al.) 2024) addresses this by introducing soft-linear blending regression losses during pretraining, improving text-motion alignment. However, such regression objectives conflict with cross-entropy: blending encourages smooth interpolations, whereas
3

---

Published as a conference paper at ICLR 2026
3.1. Autonnatic actin-dsseripdion onnotation pipeline
A2. Motion resnstruction
Epcome
PCST (Parale Coah (hongly propgo
ciol-innl Assotition
Heuritic fitrring
Suage Openereccio High-lerel arresaatoo
Fineatninet Anssetttnt
High
sasars linng Minii
with
Pansin
Chastiing
valo wdiing siles
Stmppnd
Figure 3: Data annotation pipeline: We generate motion-text pairs from egocentrie videos using novel automated annotation framework combined with a state-of-th oftte-o--thh-rtttt hand tracker. Text annotations produced by first applying Parallel followed by a voccbuloyc refinement stage.
prompting for open- gpen-vocabulary reasoning,
CE enforces sharp token choices, leading to ambiguous representations and reduced generalization. Our approach extends this line of work with a geometry -alintee stage after pretraining, where Gumbel-Softma sampling and reconstruction losses guide the LLM toward motions that are both semantically grounded and geometrically consistent.
VOVAE Motion Prior: VQ-VAE tokenizers discretize motion into language like symbols (Jiang ct al. 2024| Guo et al.) 2022). but single codebooks fail to capture multi modality. Extensions use multiple codebooks: for hand/face (Yi et al. (023)), hand/object (Huang al. 2025), decomposed
body parts (Chen et al.] 2024). (Wang et al. (025) further explore scaling strategies to expand
capacity. We build on these ideas by disentangling trajectories and hand poses into distinct codebooks, and further separate left and right hands. This yields finer control and improved generalization under temporal compression, surpassing prior single- and multi-codebook designs. A more detailed version of the related works is presented in Appendix D.
3 3D HANDS IN THE WILD (3D-HIW) DATASET
To enable till hand motion modelling, we construct a large-scale 3D hand motion dataset based on in-the-wild videos from Ego4D Grauman et al. (2022) and Ego Vid5M Wang al. (2024). We propose a two-stage VLM--aseed text annotation and a motion reconstruction pipeline.
3.1 AUTOMATIC TWO-STAGE TEXT ANNOTATION PIPELINE
To generate textual descriptions from egocentric action videos, we propose an automated two-stage annotation pipeline using VLMs/LLMs. We employ VILA (Wu et al., (024) as the VLM for its strong performance in vidco-language understanding and scalability for dense frame-level queries. Generating reliable annotations from egocentric videos is complicated, since the model needs to jointly reason about hand motion, user intent, and object scene relationships. To address these challenges, we propose a two-stage pipeline, shown in Figure 3 In Stage (Open-vocabulary high- level annotation), introduce Parallel Chain-of-Thouuh prompting strategy, which decomposes the reasoning process into several atomic prompts focused on the hand role, action-object relations, state transitions, and intent. These responses are then aggregated by summarization LLM (Claude) to produce a coherent high-level description and reduce ballucinations. In Stage 2 (Closed-vocabulae fine-grained annotation), we refine these high-level annotations by constraining the VLM to select plausible object-and -ajicotactionn pairs from curated vocabulary, mined from Ego and Ego4D narrations and organized into semantically meaningful clusters. This closed-vocabulary grounding improves consistency, and yields more faithful fine-grained annotations. We present the annotations generated by our method for a few sample sequences in Figure 5 Finally, we verify the generated annotations using an additional VLM pass and filter outliers with a Local Outlier Factor (LOF) filter. These refined annotations serve as supervision for the downstream training of our text noxt-conditioned hand motion synthesis model. The prompts used for the annotation pipeline presented in Appendix C.3
4

---

Published as a conference paper at ICLR 2026
Stage I: Open-vocabulary high-level annotation
Stage 2: Closel-vocabulary fine-grained aunotation
Atomic prompity and repooves (Ponallel CoT):
af isputs:
Head role: Right land maiislas loik, lef land subilizes the locad brad
-Sage= uimmrry: 'The person spreads bread using o kniie""
AAtlon abject relatien: Hladd ren- lnis spood
Polcatial sbjeck: (breac aisce, anide, buster, plate, cutting poard]
Stase tranition: Sccep - spread - place dicee -
oepoat on starher eiic..
Potcatial (acoup, apennd, holld/tabiilio. plase, pick up)
heporing
propaatioo))
Poteatisd
by Summarization (Claale)::
6) Refinement (Clawde):
"The person spreads butter on bread slices using a kniff."
right while the left hand holds the siice steady.'
knife sceap spread
Figure 4: Example of the two-stage annotation pipeline for an egocentric video (Figure
8GB beages
High-level caption: persos troee biorte
High level caption:: The person aperads butter on bevad aloes Fine-gralzed cupéions The pezeo sas taei riats basd apread butter on salor of then places bestered boaad oo pivit They repest oction moother honed.
Finegriii caption: pemse srriil fins shile Uslr smooth the Sibre remorigg any venicke crman..
with ereating
operato
Figure 5: Examples of the generated annotation and motion reconstruction from egocentric videos using our data annotation pipeline. For better visulaization please see SupMat video.
3.2 MOTION RECONSTRUCTION
To extract 3D hand motion reconstructions from egocentric videos, we first process high-level text descriptions from the Ego dataset identify sequences involving human presence, particularly those where humans interact with objects. We then cluster these textual descriptions into scene-level activity categories (e.g., crafting, repair) and sample sequences from each cluster to ensure diverse coverage, given that certain categories like cooking are overrepresented. Next, we run a hand keypoint tracker over the sampled vidcos and retain only those seguences where both hands are visible in at least 80% of the frames. We use HaWor (Zhang et al. 2025a) to reconstruct 3D hand motions from these egocentric sequences in a global coordinate frame. To reduce the noise in the reconstructed motions, apply the Savitzky- Finally, we compute the mean of the top-3 sequence-level acceleration on both translation and rotation parameters identify and filter out samples with abrupt, transitions, indicating Ha Wor failures.
filter (Savitzky & Golay (9644) followed by Gaussian filter.
3.3 DATASET ANALYSIS
Our '3D Hands in the wild' (3D-HIW) motion dataset contains 5000 minutes of 3D hand poses and text descriptions, covering over 1355 objects and 1045 verbs. In total, 3D-HIW comprises 12M hand poses represented with MANO parameters. In Figure we compare the top-200 trajectories between 3D-HIW and mocap datasets. While captured motions appear repetitive and front- fantin in-the tn-the-wild motions show greater variability in shape, end positions, and speed. t-SNE embeddings of trajectories and hand poses of top-3000 diverse samples further confirm that 3D-HIW spans a broader distribution than GRAB Gigahands, capturing richer variability of real-world interactions. For more details, see Appendix C.1
4 MOTION MODELLING
To model in-the-wild hand motions, we first tokenize the motion space into discrete tokens using a decomposed (Q-VAEB.aar in unified latent space which allows us to do both motion synthesis from text and captioning of hand motions. Motion parameterization: We represent the hand motions as M (H,, Hr) E IR DxN where N represents the total number of frames in the motion, D represents the motion dimension, and 1/r denotes the left and right hand respectively. The hand motions are parameterized using the MANO hand model (Romero et al. 2017) represented as H. = (r),0,) E D/2xNN with j E {l, r). Tj 8 represents the trajectory of the hand motion, which contains 6D global rotation and translation. 0g N denotes the hand pose representing the 15 joints with 6D rotation.
this motion space, train an LLM to model text and motion tokens
5

---

Published as a conference paper at ICLR 2026
Oigatiak
Stat
-Oiga
30.019 (Oun))
CIAAD Gigallieds
Tojenny Dieviuzice
Foet Dustitation
Figure 6: Comparison of our 3D-HIW dataset with existing datasets (GRAB, Gigahands). Left: 2D trajectory density plots show that dataset covers broader spatial range with more diverse start-end distributions. Right: t-SNE embeddings of trajectories and hand poses further highlight that our data spans significantly wider distribution, capturing natural variability.
4.1 STRUCTURING HANDS INTO FINE-GRAINED TOKENS (SHIFT):
Standard VQ-VAE models struggle to capture the diversity and complexity of 'in-the-wild' hand motion, often resulting in limited recon- struction quality and generalization. To address this, we introduce SHIFT tokenizer that mod- els trajectory and pose components using sep- arate VQ- VAEs, while also disentangling left and right hands during encoding and decoding. This design choice is motivated by prior findings from Huang et al. 2025); Chen al. (2024). where separating motion into different parts like
Trajectory codebook
Enceder
Trejeolory
Trijetory Decoler
HH
2
R
Ga 22.6
Hand- Hand-pose pose codebook
sppooura avod pond
pose Desnder
6
Vr: Leftt///Right Hand
(2
Figure 7: SHIFT Tokenizer overview.
hand, face, and objects shows improved performance. Our work extends this idea further by sep- granular components. Empirically, this formulation achieves stronger generalization and more faithful reconstructions (Tablef4, even under high temporal compression (Figure 1Op.. The hand motions encoded using trajectory E, and hand pose Eg encoders to produce 2j E dx and Vj dR N/8 embeddings, where d represents the dimension of the codebook latent space. The embeddings are quantized into 2, and Ū3 using nearest neighbor quantization (van den Oord et al. 2018). The trajectory f, and hand pose f of the input sequence reconstructed using the respective decoders D, and hand pose Dp. to get the final reconstructed motion A = (fj. 0,) we train the encoder, decoder, and codebook simultancously with the loss:
arating the motion into part-modality-speee
Lvq = (re M) + Σ (1sg[a)] TE
a||2 B|| sg[$]]13. X {21, 2r, 31. yr
where Cree an MSE reconstruction loss, sg is a stop gradient operation used to calculate the codebook loss, and the third part is a "commitment" loss with trade-off B.
4.2 LLM FOR HAND-MOTION MODELLING:
Employing the part-modality decomposed tokenizer, a hand motion sequence MI.N can be mapped discrete trajectory and pose tokens Z1:7 = and y1.r = {ye) We represent the motion tokens as sequences of indices S1.27 = {st ) E N, where each is drawn from the combined motion vocabulary space Vn, where trajectory and pose codebooks are stacked. When tokenized, the motion sequence is represented each motion token is written as a special symbol <mot smotion.token( motion tokens as (m) and text tokens as (t).
interleaved stream of trajectory and pose tokens. In practice,
For brevity, we denote
For example, sequence with T trajectory (m(r) and pose tokens (m (0)) is arranged as:
(som) (mi (Tx.)
(m (mm))m))mm)))))) (m)
(TR)) (m) (0n) ..... (m (Ta)) (m (8x.) (mm (m (0n)) (com).
(Tn)
(2)
To train the LLM, we build a unified eext-motion space V = V₁ Vm. where Vú2 is the text vocabulary. We include additional special tokens such as boundary markers (e.g., <som>, <eom>), which enable
text-conditioned motion tasks to be represented in a consistent format. The model handles text- to-motion, motion-to-text, or joint captioning tasks in unified manner. Given input sequence

---

Published as a conference paper at ICLR 2026
X, (ag 23 V, predicts the target sequence X, f2r of E V autoregressively:
Po Xx)= = II
po X.)
(3)
==0
The training objective is:
LLM Σ
log po (2) x'exs X.).
(4)
Pre-training Stage. We pre-train the language model on large-scale text and motion sequences using a cross-entropy loss on the next-token-pret next-token-prediction task and simple T2M and M2T tasks. This allows the model to capture natural language semanties and temporal dynamies of hand motions, similar to MotionGPT.
Crometric-Refinemeee Stage. While token-level cross-entropy loss encourages correct next-token prediction, we find it does not guarantec that decoded motions geometrically smooth or realistic. Prior works (Hong et al., 2024) address this by adding soft-blending-base regression losses during the pre-training stage. However, jointly applying softblenddingbbsee regression in pre-training conflicts with cross-entropy, as soft-blending favors smooth interpolations while CE enforces sharp token predictions, leading modest performance improvements (Table 5). To address this, we adopt Gumbel-Softmal parameterization, which enables discrete token selection while directly applying regression loss in motion space. This yields the joint training objective: L aLLAS where Lree ensures fidelity of the reconstructed hand motion. In addition, we also train the model on additional masked prediction tasks with a encourage the model to focus more on the reconstruction quality.
Cree-
Instruction Fine-tuning Stage. Finally, we perform instruction fine-tuning to enable the model to handle multiple tasks, including text-to-motion and motion-to-text. We adopt the multi-task prompt-based training strategy from MotionGPT, where the model is supervised on diverse instruc- tion prompts. This stage improves generalization across different tasks and yields state-of-the-an performance on both synthesis and captioning benchmarks.
EXPERIMENTS
Dataset We build our experiments on the proposed 3D-HIW hand motion dataset, which provides paired 3D hand motions and text descriptions of 32k real-world sequences. For training and evaluation, we partition the sequences into oonoolllapping splits avoid leakage between sets. Specifically, we allocate 80% % for training (26k sequences), 10% for validation (3k), and 10% for testing (3k).
Evaluation Metrics: For text-to-motion generation (T2M), we follow prior work Tevet et al. (2023) Guo al. (2022) and report: R Precision (RP3) for text-motion matching. MMD for text and motion alignment in feature space, KID for distribution similarity, and Diversity for output variability and Multimodality for diversity from single prompt. For motion-to-text captioning, we use standard language metrics (BLEU4, BLEEUI, Rouge-L) along with R Precision. For annotation quality, we adopt GPT-Score following EgoHOD Hong al. (2024). For motion reconstruction, we report MPJPE, PA-MPJPE, and ACCEL as in E et al. (2024).
5.1 DATASET ANNOTATION
We evaluate the quality of our egocentric video-to-text annotations using GPT-Scores from the EgoHOD (Pei et al. 2025), which rate similarity to human-authored descriptions on a 0-10 scale (higher is better). Results are reported in Table 3 Compared LaVILA (Zhao et al.) 2023) and EgoHOD, our method achieves the highest GPT-Score (6.9), surpassing existing approaches by a clear margin. This confirms that our pipeline produces more faithful and higher-quality text annotations. To further analyze the role our two-stage annotation pipeline, we ablate against two baselines: (i) VILA- VILA-Naive, which uses single large prompt, and (ii) VILA-Stagel, which only uses the first-stage
7

---

Published as a conference paper at ICLR 2026
Table 1: Comparison of various methods on RPrecision, MMDist, KID Mean, Diversity, and
MultiModality, Lower is better for all metrics except RPrecision and Diversity.
Method
RP3
MMD 1
KID 4
Div +
MM
Ground Truth
0.667 = 0.004
1,903 = 0.005
3.964 2 0.189
HumanMDM
1.971 0.019
0.344 = 0.02
3.177
1.748 0.069
MotionGPT
0.573 0.009
2.183
0.756
3.642
2.015 0.095
T2M-GPT
0.683 0.005
1.976
0,431 = 0.02
0.134
1,892 0.085
Ours
0.721 0.004
1.765 # 0.016
0.216 + 0.02
3.865 + 0.124
1,984 = 0.084
outputs. Both underperform compared to our full pipeline. validating the importance of structured multi-stage prompting for robust annotation quality. We study motion quality of the 3D-HiW dataset with respect to different data-cleaning steps Appendix C.1
Table 3: Evaluating text annotations using
Table 2: results.
Motion-to-text captioning quantitative
EgoHoD's s GPT-Scores (0-10).
Method
GPT-Score
Method
RP3 1
B4 f
B1 1
RG
LaVILA
4.9 0.3
GT
0.668
EgoHOD
6.1 0.4
TM2T
0.385
0.122
0.333
0,428
MotionGPT
0.407
0.132
0.345
0.439
ViLA-Naive
0.2
VIL.A-Staell
6.4 0.5
Ours
0.571
0,181
0.420
0,472
Ours 6.9 0.3
5.2 CLUTCH- TEXTI---TT--OO GENERATION (T2M)
The text-to-motion task evaluates model's ability to generate plausible hand motion sequences given natural language input. We benchmark CLUTCH against recent state-of-the-art baselines. including MotionGPT (Jiang et al. 2024), HumanMDM TTevet et al. 2023), and T2MGPT (Zhang)
et al. 2023), retraining all models on our dataset for fairness. Results are reported in Table] [ Across all metrics, CLUTCH achieves consistent improvements over competing methods, suggesting that its unified modelling of language and hand motion provides stronger alignment than prior approaches. Qualitative results in Figure 8 further highlight CLUTCH's ability to generate multiple diverse yet semantically faithful motion trajectories from the same textual description.
lopuct promape person press key disstronie kaybosra vousie"
proutpt + "Re poras, kuiie, fabriz"
Bouhh Roah
Figure 8: Qualitative results for text-to-motion synthesis.
5.3 CLUTCH- MOTION-TO-TT--AA CAPTIONING (M2T)
The motion-to-tens task involves generating text descriptions from novel 3D hand motions from the wild. To this end, we compare our method against MotionGPT and TM2T (Guo et al., 2022) and report the metrics in Table _ From the results, the baselines on all the metrics. We show qualitative results of motion captioning in Figure
infer that method significantly outperforms
5.4 ABLATIONS
Effectiveness of the SHIFT tokenizer: We compare our SHIFT with three baselines: MotionGPT's VQ-VAE, a standard VQ-VAE, and a part-decomposed variant (PD VQ-VAE) that disentangles left and right hands during encoding and decoding. As shown in Table 4 our model achieves the best overall performance, yielding the lowest MPJPE (45.94) and ACCEL (5.395), while also improving motion diversity. Moreover, Figure[TOilllutraae, that SHIFT handles temporal compression substantially better than the baseline VQ-VAÉs, enabling L.M training under modest memory requirements (4 A100 GPU's vs 64 Tesla V V100 and 32 NVIDIA A100 GPU's in MotionGPT and HoiGPT respectively). These results underscore the advantage of decomposing both body parts and modalities in VQ-VAE-based motion modelling. Additional experiments are presented in Appendix B.2.
Impact of Geometric Refinement and Instruct-Fine Tuning: Table compares different training stages. Pre-trane Pre-training alone (row 1) provides a reasonable baseline, but performance remains limited. Instruction tuning (IFT) substantially improves results (row w/o GR), raising T2M RP3 from 0.53 to
8

---

Published as a conference paper at ICLR 2026
Impur "hhe persoo przzz key ave clectronic kzybtard ortete musiis?
soput pronit: The person knits foérie"
Rosth
Mosuil
Rosii 2
Revuh
Figure 9: Motion-to- Text captioning results.
Table 4: Comparison of VQ-VAE configurations.
Figure 10: VQ-VAE compression.
Method
Num. / dim
MPJPE
ACCEL
Div
GT
3.964
MotionGPT
5127512
93.486
8.340
3.683
Sild wattt
Mellaac AuI
Std. VQ-VAE
4K/64 64
93.258
7.771
3,450
PD VQ-VAE
4K/64
95.266
7.500
3.647
Ours
4K764
45.944
5.395
3.747
0.69 and M2T RP3 from 0.50 to 0.57. Adding geometric refinement (GR) further boosts alignment: the full model (PT+GR+IFT) achieves the lowest KID (0.216 0.297 w/o GR) and the highest RP3 scores (0.72 for T2M, 0.57 for M2T). This demonstrates that GR plays key role in motion synthesis quality. In other words, IFT scales generalization, while GR makes that generalization meaningful by enforcing geometric alignment. The combination yields the best overall performance. Finally, we compare against the EgoLM Hong et al. (2024) poft-blending reconstruction loss (last row). While competitive, it is inferior to our approach, highlighting the benefits of explicit geometric refinement and Gumbel-Softmax based reconstruction.
Impact of Dataset Size: Increasing the number of captioned sequences from 7K to 30K yields steady improvements in both text-to-motion (T2M) and motion-to-text (M2T). These results underline the importance of larger, more diverse training data for scalable in-the-wild hand motion modelling. For reference, we also provide our method trained on a combination Arctic and GRAB dataset.
Impact of Language Model Size: Table re- ports the effect of scaling the backbone language model from T5-Small to T5-Large. As expected, larger models yield consistently better results on both T2M and M2T tasks. These results confirm that language model capacity plays a crucial role enabling stronger generalization across modalities in both tasks.
Table 7: Impact model size on the performance.
T2M
M2T
Method
RP3 1
KID L
RP3
B4
TS- Small (50M)
0.545
0.732
0.292
0,089
TS- BS-Base (220M)
0.721
0.216
0.571
0.181
TS-Large (770M)
0.733
0.092
0.578
0.192
CONCLUSION
To the best of our knowledge, CLUTCH the first work to explore in-the-wild hand motion modelling. While effective, approach still has limitations. We focus hand motions, while leaving hand-object interactions for future exploration due to the current challenges of in-the- wild reconstruction. Further improvements may enhance fine-grained expressiveness in motion reconstructions and enable temporal segmentation of overlapping actions in egocentric sequences. Advancing along these directions could further improve dataset quality and model robustness.
Table 5: Impact of different L.MM training stages. Table 6: Performance scaling with increased PT: Pre-training, GR: Geometry refinement, IFT: training data (7K, 15K, 30K samples). Cap. Instruct Fine-tuning.
data: Artic+GRAB.
T2M
M2T
T2M
M2T
Method
RP3
KID L
RP3
B4 f
Method
RP3
KID 1
RP3 1
PT w/o GR (1+IFT)
0.533
0.349
0.501
0.148
Cap. data
0.097
1.970
0.083
0.004
0.690
0.297
0.568
0.173
7K
0.513
0.860
0.247
0.092
PT GR IFT
0.721
0.216
0.571
0.181
15K
0.637
0.672
0.396
0.139
EgoLM setup
0.705
0.263
0.570
0.171
30K
0.721
0.216
0.571
0.181

---

Published as a conference paper at ICLR 2026
Despite these challenges, CLUTCH makes important progress towards scalable, natural hand motion synthesis. To this end, we introduce a novel data annotation pipeline, a dataset, and a part-modality decomposed VQ-VAE for in-the-wild hand motion modelling. Through detailed experiments, demonstrate that CLUTCH outperforms existing diffusion and L.LM models on the in-the-wild hand motion modelling task. Looking ahead, believe combining in-the-wild motions with controlled datasets, and extending to hand-object interactions can unlock new downstream applications in behavioral AI, allowing us eventually build embodied avatars capable of fine-grained high-fidelity interactions with their environments.
7 ACKNOWLEDGMENT
Justus Thies is supported by the DFG Excellence Strategy EXC-3057 ("Reasonable Artificial Intelligence". Project No. 533677015) and the project is co-funded by the European Union (ERC, Lemo, 101 162081). Views and opinions expressed are, however, those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Couneil. Neither the European Union nor the granting authority can be held responsible for them. Gerard Pons-Moll is supported by the Carl Zeiss Foundation, Amazon, MPI Science Hub and the Deutsche Forschungs- gemeinschaft (DFG, German Research Foundation) under Grant No. 409792180 (Emmy Noether Programme, project: Real Virtual Humans). He is further supported by the German Federal Ministry of Education and Research (BMBF) through the Tübingen AI Center (FKZ: 01IS (IIS18039A). Gerard Pons-Moll is a member of the DFG Cluster of Excellence "Machine Learning - Science" (EXC 2064/1, Project No. 390727645). Further, we also thank B. Kabadayi, M. Diomataris, S. Tripathi and Mayilvahanan for valuable discussions.
New Perspectives for
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Nikos Athanasiou, Alpár Ceske, Markos Diomataris, Michael J Black, and Gül Varol. Motionfix: Text-driven 3d human motion editing. arXiv preprint arXiv:2448.0077 00712, 2024.
Junuk Cha, Jihyeon Kim, Jae Shin Yoon, and Seungryul Back. Text2hoi: Text-guided 3d motion generation for hand-object interaction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1577-1585, 2024.
Yu-Wei Chao, Wei Yang. Yu Xiang, Pavlo Molchanov, Ankur Handa, Jonathan Tremblay, Yashraj S. Narang, Karl Van Wyk, Umar lqbal, Stan Birchfield, Jan Kautz, and Dieter Fox. DexYCB: benchmark for capturing hand grasping of objects. In IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2021.
Changan Chen, Juze Zhang, Shrinidhi Kowshika Lakshmikanth, Yusu Fang. Ruizhi Shao, Gordon Wetzstein, Li Fei-Fei, and Ehsan Adeli. The language of motion: Unifying verbal and non-verbal language of 3d human motion. In arXiv, 2024.
Seunggeun Chi, Hyung-gun Chi, Hengho Ma, Nakul Agarwal, Faizan Siddiqui, Karthik Ramani, and Kwonjoon Lee. M2d2m: Aulti-motion generation from text with discrete diffusion models. arXiv preprint arXiv:2407. 14502, 2024.
Sammy Christen, Shreyas Hampali, Fadime Sener, Edoardo Remelli, Tomas Hodan, Eric Sauser, Shugao Ma, and Bugra Tekin. Diffh2o: Diffusion- synthesis of hand-object interactions from textual descriptions. In SIGGRAPH Asia, pp. 145:1-145: 2024. URL httpp:://dot org/
10.1145/368035 0011///6680228888555588
Zicong Fan, Omid Taheri, Dimitrios Tzionas, Muhammed Kocabas, Manuel Kaufmann, Michael J. Black, and Otmar Hilliges. ARCTIC: dataset for dexterous bimanual hand-object manipulation. In Proceedings IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2023.
Rao Fu, Dingxi Zhang, Alex Jiang, Wanjia Fu, Austin Fund, Daniel Ritchie, and Srinath Sridhar. Gigahands: A massive annotated dataset of bimanual hand activities. 2025.
Anindita Ghosh, Rishabh Dabral, Vladislav Golyanik, Christian Theobalt, and Philipp Slusallek. Imos: Intent-driven full-body motion synthesis for human-object interactions. In Eurographics, 2023.
Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Gird- har, Jackson Hamburger. Hao Jiang, Miao Liu, Xingyu Liu, Miguel Martin, Tushar Nagarajan, Ilija Radosavovic, Santhosh Kumar Ramakrishnan, Fiona Ryan, Jayant Sharma. Michael Wray, Mengmeng Xu, Eric Zhongcong Xu, Chen Zhao, Siddhant Bansal, Dhruv Batra, Vincent Car- tillier, Sean Crane, Tien Do, Morrie Doulaty, Akshay Erapalli, Christoph Feichtenhofer, Adriano Fragomeni, Qichen Fu, Abrham Gebreselasie. Cristina Gonzalez, James Hillis, Xuhua Huang, Yifei Huang, Wenqi Jia, Weslie Khoo, Jachym Kolar, Satwik Kottur, Anurag Kumar, Federico Landini, Chao Li, Yanghao Li, Zhenqiang Li, Karttikeya Mangalam, Raghava Modhugu, Jonathan Munro, Tullie Murrell, Takumi Nishiyasu, Will Price, Paola Ruiz Puentes, Merey Ramazanova, Leda Sari, Kiran Somasundaram, Audrey Southerland. Yusuke Sugano, Ruijic Tao, Minh Vo. Yuchen Wang, Xindi Wu, Takuma Yagi, Ziwei Zhao, Yunyi Zhu, Pablo Arbelaez, David Crandall, Dima Damen, Giovanni Maria Farinella, Christian Fuegen, Bernard Ghanem, Vamsi Krishna Ithapu, C. V. Jawahar, Hanbyul Joo, Kris Kitani, Haizhou Li, Richard Newcombe, Aude Oliva, Hyun Soo Park, James M. Rehg, Yoichi Sato, Jianbo Shi, Mike Zheng Shou, Antonio Torralba, Lorenzo Torresani, Mingfei Yan, and Jitendra Malik. Ego4d: Around the world in 3,000 hours of egocentric video. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 18973-18990, 2022.
Chuan Guo, Xinxin Zuo, Sen Wang, and Li Cheng. Tm2r: Stochastic and tokenized modeling for the reciprocal generation of 3d human motions and texts. In ECCV, pp. 580-597. Springer, 2022.
Chuan Guo, Yuxuan Mu, Muhammad Gohar Javed, Sen Wang. and Li Cheng. Momask: Generative masked modeling of 3d human motions. In CVPR, pp. 1900-1910, 2024.
11

---

Published as a conference paper at ICLR 2026
Ikhsanul Habibie. Mohamed Elgharib, Diogo Luvizon, Balamurugan Thambiraja, Simbarashe Nyat- sanga, Justus Thies, Michael Neff, and Christian Theobalt. Comand: Controllable action- aawre manifold for 3d motion synthesis. In International Symposium on Vision, Modeling, and Visualiza- fion. Eurographics Association, 2024.
Fangzhou Hong, Vladimir Guzov, Hyo Jin Kim, Yuting Ye, Richard Newcombe, Ziwei Liu, and Lingni Ma. Egolm: Multi-modal language model of egocentric motions. arXiv preprint arXiv:2409. 18/27, 2024.
Mingzhen Huang. Fu-Jen Chu, Bugra Tekin, Kevin Liang, Haoyu Ma. Weiyao Wang. Xingyu Chen, Pierre Gleize, Hongfei Xue, Siwei Lyu, Kris Kitani, Matt Feiszli, and Hao Tang. Hoigpt: Learning long sequence hand-object interaction with language models. In IEEE Conference on Computer Vision and Puttern Recognition (CVPR), Nashville, USA, 2025.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparametriizaioon with gumbel-softmat In Proceedings International Conference on Learning Representations (ICLR), April 2017. URL
httpp:://ppenreieew.neeee
Biao Jiang, Xin Chen, Wen Liu, Jingyi Yu, Gang Yu, and Tao Chen. Motiongpt: Human motion foreign language. Advances in Neural Information Processing Systems, 36, 2024.
Biao Jiang, Xin Chen, Chi Zhang, Fukun Yin. Zhuoyuan Li, Gang Yu, and Jiayuan Fan. Motionchain: Conversational motion controllers via multimodal prompts. In ECCV, pp. 54-74. Springer, 2025.
Korrawe Karunratanakul, Konpat Preechakul, Supasorn Suwajanakorn, and Siyu Tang. Guided motion diffusion for controllable human motion synthesis. In ICCV, pp. 2151-2162, 2023.
Taein Kwon, Bugra Tekin, Jan Stühmer, Federica Bogo, and Marc Pollefeys. H2o: Two hands manipulating objects for first person interaction recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 10138-10148, October 2021.
Chuqiao Li, Julian Chibane, Yannan He, Naama Pearl, Andreas Geiger, and Gerard Pons-Moll. Unimotion: Unifying 3d human motion synthesis and understanding. In International Conference on 3D Vision (3DV), March 2025a.
Muchen Li, Sammy Christen, Chengde Wan, Yujun Cai, Renjie Liao, Leonid Sigal, and Shugao Ma. Latenthoi: On the generalizable hand object motion generation with latent hand diffusion. In Proceedings of the Computer Vision and Patlern Recognition Conference (CVPR), pp. 17416- 17425, June 2025b.
Haiyang Liu, Zihao Zhu, Giorgio Becherini, Yichen Peng, Mingyang Su, You Zhou, Xuefei Zhe, Naoya Iwamoto, Bo Zheng, and Michael J. Black. Emage: Towards unified holistic co-speech gesture generation via expressive masked audio gesture modeling. In CVPR, 2024.
Jinpeng Liu, Wenxun Dai, Chunyu Wang, Yiji Cheng, Yansong Tang, and Xin Tong. Plan, posture and go: Towards open-world text-to-motion generation. arXiv preprint arXiv:2312. 14828 2023,
Naureen Mahmood, Nima Ghorbani, Nikolaus F Troje, Gerard Pons-Moll, and Michael Black. Amass: Archive of motion capture as surface shapes. In ICCV, pp. 5442-5451, 2019.
Evonne Ng, Javier Romero, Timur Bagautdinov, Shaojie Bai, Trevor Darrell, Angjoo Kanazawa, and Alexander Richard. From audio photoreal embodiment: Synthesizing humans in conversations, 2024.
Baoqi Pei, Yifei Huang, Jilan Xu, Guo Chen, Yuping He, Lijin Yang, Yali Wang, Weidi Xie, Yu Qiao, Fei Wu, and Limin Wang. Modeling fine-grainet hand-object dynamics for egocentric video representation learning, 2025.
Ilya A. Petrov, Riccardo Marin, Julian Chibane, and Gerard Pons-Moll. Tridi: Trilateral diffusion of 3d humans, objects, and interactions. In Proceedings of the IEEE/CVF International Conference Computer Vision October 2025.
12

---

Published as a conference paper at ICLR 2026
Adam Roberts, Hyung Won Chung, Anselm Levskaya, Gaurav Mishra, James Bradbury, Daniel Andor, Sharan Narang, Brian Lester, Colin Gaffney, Afroz Mohiuddin, Curtis Hawthomne, Aitor Lewkowyez, Alex Salcianu, Mare van Zee, Jacob Austin, Sebastian Goodman, Livio Baldini Soares, Haitang Hu, Sasha Tsvyashchenko, Aakanksha Chowdhery, Jasmijn Bastings, Jannis Bulian, Xavier Garcia, Jianmo Ni, Andrew Chen, Kathleen Kenealy, Jonathan H. Clark, Stephan Lee, Dan Garrette, James Lee-Thorp, Colin Raffel, Noam Shazeer, Marvin Ritter, Maarten Bosma, Alexandre Passos, Jeremy Maitin-Shepard, Noah Fiedel, Mark Omernick, Brennan Saeta, Ryan Sepassi, Alexander Spiridonov, Joshua Newlan, and Andrea Gesmundo. Scaling up models and data with 5x and seqio. arXiv preprint arXiv:2203.171588 2022. URL htttps://aaxxt
org abs//ab//a0 2203. 17189
Javier Romero, Dimitrios Tzionas, and Michael J. Black. Embodied hands: Modeling and capturing hands and bodies together. ACM Transactions on Graphics, (Proc. SIGGRAPH Asia), 36(6), November 2017.
A Savitzky and MJE Golay. Smoothing and differentiation of data by simplified least squares procedures. Analytical Chemistry, 36(8):1627-1639, 1964.
Yonatan Shafir, Guy Tevet, Roy Kapon, and Amit H Bermano. Human motion diffusion as a generative prior. arXiv preprint arXiv:2303.01418. 2023.
Connor Shorten, Charles Pierse, Thomas Benjamin Smith, Erika Cardenas, Akanksha Sharma, John Trengrove, and Bob Luijt. Structuredrag: JSON response formatting with large language models. CoRR, abs/2408 1061, 2024. doi: 10.48550/ARXW.248.. 1066. URL https Aydoi.org// 48
2408 11061
Vanessa Sklyarova, Egor Zakharov, Otmar Hilliges, Michael J Black, and Justus Thies. Haar: Text-conditioned generative model of 3d strand-based human hairstyles. ArXiv, Dec 2023.
Omid Taheri, Nima Ghorbani, Michael J. Black, and Dimitrios Tzionas. GRAB: A dataset of whole- body human grasping of objects. In European Conference Computer Vision (ECCV), 2020. URL http:t:tttggaaaaaae http:/ggabb... is tue. mpg. de
Guy Tevet, Sigal Raab, Brian Gordon, Yoni Shafir, Daniel Cohen-or, and Amit Haim Bermano. Human motion diffusion model. In The Eleventh International Conference on Learning Represen- tations, 2023.
net/ wwrumrrruuieoooo SttrrummiddSSJkkSSyy jww
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, 2018.
Xiaofeng Wang. Kang Zhao, Feng Liu, Jiayu Wang, Guosheng Zhao, Xiaoyi Bao, Zheng Zhu, Yingya Zhang, and Xingang Wang, Egovid-5m: A large-scale video-action dataset for egocentric video generation. arXiv preprint arXiv:2411 1008380, 2024.
Ye Wang, Sipeng Zheng, Bin Cao, Qianshan Wei, Weishuai Zeng. Qin Jin, and Zongqing Lu. Scaling large motion models with million-level human motions. In Forty- secoy- International Conference on Machine Learning, 2025. URL https::/ net forummidd
TO6 TO65rwuxxt
Zhenzhi Wang, Jingbo Wang. Dahua Lin, and Bo Dai. Intercontrol: Generate human motion interactions by controlling every joint. arXiv preprint 1rXiv:2311.15860 2023.
Tsung-Han Wu, Heekyung Lee, Jiaxin Ge, Joseph E Gonzalez, Trevor Darrell, and David M Chan. Generate, but verify: Reducing hallucination in vision-language models with retrospective resam- pling. arXiv preprint arXiv:2504.1m 13169, 2025.
Yecheng Wu, Zhuoyang Zhang, Junyu Chen, Haotian Tang, Dacheng Li, Yunhao Fang, Ligeng Zhu, Enze Xie, Hongxu Yin, Li Yi, et al. Vila-u: a unified foundation model integrating visual understanding and generation. arXiv preprint arXiv:2409.0444
2024.
Yiming Xie, Varun Jampani, Lei Zhong, Deqing Sun, and Huaizu Jiang. Omnicontrol: Control any 2023.
joint any time for human motion generation. arXiv preprint
13

---

Published as a conference paper at ICLR 2026
Lixin Yang, Kailin Li, Xinyu Zhan, Fei Wu, Anran Xu, Liu Liu, and Cewu Lu. OakInk: A large-scale knowledge repository for understanding hand-object interaction. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Hongwei Yi, Hualin Liang, Yifei Liu, Qiong Cao, Yandong Wen, Timo Bolkart, Dacheng Tao, and Michael Black. Generating holistic 3d human motion from speech. In CVPR, pp. 469-480, 2023,
Xinyu Zhan, Lixin Yang, Yifei Zhao, Kangrui Mao, Hanlin Xu, Zenan Lin, Kailin Li, and Cewu Lu. Oakink2: dataset of bimanual hands-object manipulation in complex task completion. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR),
pp. 445-456, June 2024.
Jianrong Zhang, Yangsong Zhang, Xiaodong Cun, Yong Zhang, Hongwei Zhao, Hongtao Lu, Xi Shen, and Ying Shan. Generating human motion from textual descriptions with discrete representations. In CVPR, pp. 14730-14740, 2023.
Jinglei Zhang, Jiankang Deng, Chao Ma, and Rolandos Alexandros Potamias, Hawor: World-space hand motion reconstruction from egocentric videos. arXiv preprint arXiv:2501.00 022973,, 2025a.
Wanyue Zhang, Rishabh Dabral, Vladislav Golyanik, Vasileios Choutas, Eduardo Alvarado, Thabo Becler, Marc Habermann, and Christian Theobalt. Bimart: A unified approach for the synthesis of 3d bimanual interaction with articulated objects. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025b.
Zhikai Zhang, Yitang Li, Haofeng Huang, Mingxian Lin, and Li Yi. Freemotion: Mocap-free human motion synthesis with multimodal large language models. In ECCV, pp. 403-421. Springer, 2025c.
Yue Zhao, Ishan Misra, Philipp Krähenbühl, and Rohit Girdhar. Learning video representations from large language models. In CVPR, 2023.
Keyang Zhou, Bharat Lal Bhatnagar, Jan Eric Lenssen, and Gerard Pons-Moll. Toch: Spatio-temporal object-to-hand correspondence for motion refinement. In European Conference Computer Vision (ECCV). Springer, October 2022.
Keyang Zhou, Bharat Lal Bhatnagar, Jan Eric Lenssen, and Gerard Pons-Moll. Gears: Local geometry-aware hand-object interaction synthesis. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2024.
14

---

Published as a conference paper at ICLR 2026
A GUMBEL-SOFTMAX MOTION DECODING.
Given an input sequence X,, the LLM outputs a full vocabulary logit tensor fx( E TRIII where [V] is the joint (text motion) vocabulary. For motion decoding, extract only the logits corresponding to the motion-token subspace Vm. C V. This slicing is expressed as:
f1r fo V
where & E RK and K |Vm is the size of the motion-token vocabulary. This corresponds exactly selecting the motion-token logit channels from the full output tensor.
The extracted motion logits are then converted into categorical representation through Gum- bel-Softmax operator (Jang et al. 2017):
7. Gumbel
The continuous 3D representation using the SHIFT decoder:
sequence reconstructed by decoding this Sonbel-Softmax motion
MI. = Dr, Do(Zi.T),
where D, denotes trajectory decoder parameters and DD the hand-pose decoder parameters.
Reconstruction Loss. To refine geometric fidelity, we combine the language-modeling loss L with reconstruction loss computed in continuous motion space:
Cree
Mr
M
T
1=1
The final objective is:
C == LLM + X Lree-
B ADDITIONAL EXPERIMENTS
Implementation details: In our experiments, we use two VQ-VAE models with 4096 codebook entries of 64 dimensions each. The compression rate the VQ-VAE is 8, i.e., the encoder compresses 8 temporal frames into single code. The motion tokenizer is trained for 2000 epochs using the Adam optimizer with learning rate of 2e 4 We employ the 220M-paramen et al., 2022) as our language model. The model is pre-trained, geometry-refined, and fine-tuned 300/50/200 /20 epochs with learning rates of 2e 2e-1/11-5/ 0 /2e--5 respectively. Experimental results are reported with 95% confidence interval, computed from 20 repeated runs to ensure statistical significance. All models are trained on 4 NVIDIA A A100 GPUs with 80GB memory each.
Flan-T5-Bas (Roberts
B. EFFECTIVENESS OF TEXT-ANNOTATI TYPE:
We evaluate how different annotation types af- fect LLM performance, using high-level (HA), fine-grained (DA), and combined (HA+DA) an- notations (Table 8h. Using only high-level (HA) or fine-grained (DA) annotations yields moder- ate performance (e.g., T2M RP3 = 0.551 and 0.462). Combining both (HA+DA) yields the best results across metrics (T2M RP3 0.721, M2T RP3 = 0.571), underscoring their comple- mentarity for robust text-motion learning.
Table 8: Effect of different types annotation on Text-to-Motion task performance. HA: High-level annotation, DA: Fine- Fine-grained grained Annotation.
T2M
M2T
Method
RP3
KID
RP3
B4
(HA)
0.551
0.148
0.496
0.153
(DA)
0.462
0.192
0.489
0.114
Ours(HA+DA)
0.721
0.216
0.571
0.181
B.2 TOKENIZER ANALYSIS:
We provide additional comparisons of our decomposed VQ-VAE (SHIFT) against several basclines to further highlight the impact of model design choices. As reported in Table 9 our formulation
15

---

Published as a conference paper at ICLR 2026
consistently achieves the lowest reconstruction error, with MPJPPE reduced to 45.94 and ACCEL to 5.395, while preserving motion diversity. Further, We visualize the effect temporal compression in Figure TO Whereas standard VQ-VAES degrade rapidly the compression factor increases, decomposition into trajectory and pose codebooks maintains reconstruction quality even at high compression rates. This property is especially important for scaling large language models to motion, as it reduces the effective sequence length and enables training under more modest compute budgets. In practice, our model requires only 4 NVIDIA A100 GPUs for training, compared the 64 Tesla V100 GPUs used in MotionGPT and 32 A100 GPUs in HOIGPT. These extended experiments confirm that decomposing both modalities (trajectory vs. pose) and body parts (left vs. right hand) is crucial factor for stable, scalable motion modeling.
Table 9: VQVAE analysis Extened version
Method
Num. / dim
MPJPE 4
ACCEL +
Div +
GT
3.964
MotionGPT
512/512
93.486
8.340
3.683
Std. VQ-VAE
4K 64
93.258
7.771
3,450
Std VQ-VAL VAE
8K 64
92.150
7.859
3.539
Std VQ-VAR
4K 256
93.045
8.014
3.647
VQ-
4K 64
95.266
7.500
3.647
VOVAE
8K 64
92.052
7.369
3,581
VQVAE
4K 256
97.289
7.616
3.357
Ours
4K 64
45.944
5.395
3.747
B.3 RESULTS ON PUBLIC DATASETS:
To further assess the capability of our method, we follow the dataset protocol of HOIGPT(Huang al. 2025) and train model and all baselines a publicly available captured dataset composed of ARCTIC et al. (203) and GRAB (Taheri et al. 2020), covering 5.1K 0.5K 0.5K sequences for training, validation, and testing. We evaluate performance on the Text2Motion (T2M) and Motion2Text (M2T) tasks using the metrics described in Section and we report the results in Table 10 and Table 11
As shown in the tables, our method consistently outperforms prior approaches across both tasks. In T2M, our model achieves the highest R-Precision (0.492), the lowest MMDist among generative models (3.008), and competitive KID scores, while also providing substantially better multimodality than MotionGPT et al. 2023), diffusion-based model, tends to generate visually smooth but less semantically aligned motions, which is reflected its lower R-Precision and higher MMDist under this reduced-daa regime. In M2T, method again achieves the best performance across all major metrics, indicating stronger bidirectional grounding between motion and language compared to MotionGPT and TM2T. Although our model explicitly designed for in-the-wild hand-motion modeling, it nonetheless generalizes effectively to controlled HOI datasets, demonstrating the strength and versatility of the learned representation.
al., 2024) and "2MGPT(Zhang et al. 2023). Notably, HumanMDM (Tevet
B.4 SENSITIVITY ANALYSIS OF THE LM AND RECONSTRUCTION LOSSES
We conducted full a/ sensitivity sweep to study the effect of balancing the
loss and the reconstruction loss. The results are presented Table 12
We observe consistent trend:
Method
RP3 1
MMDist
KID L
Diversity
MultiModality
Ground Truth
0.525
2.763
4.581
HumanMDM
0.429
4.047
0.0107
4.915
2.567
MotionGPT
0.371
3.609
0.0409
3.315
1.955
T2MGPT
0.407
3.761
0.0773
4.956
1.658
Ours
0.492
3.008
0.0144
3.811
2.393
Table 10: T2M evaluation results on ARCTIC+GRAB.
16

---

Published as a conference paper at ICLR 2026
Method
RP3
Bleu4 f
Bleul f
ROUGE L
TM2T
0.3519
0.1815
0.2245
0.5174
MotionGPT
0.4262
0.2158
0.5167
0.5278
Ours
0.4601
0.2341
0.5732
0.5822
Table 11: M2T evaluation results on ARCTIC+GRAB.
T2M RP3
M2T
LM (a:
Rec (A)
KID
RP3 1
Bleu4 f
GT
0.671
0.667
0
1
0.413
0.886
0.099
0.021
0.1
0.9
0.498
0.725
0.357
0.077
0.25
0.75
0.522
0.335
0.403
0.116
0.5
0.5
0.721
0.216
0,571
0.181
0.75
0.25
0.712
0.234
0,544
0.172
0.9
0.1
0.708
0.289
0.543
0.171
0
0.690
0.297
0.568
0.173
Table 12: Sensitivity study of the LM loss weight and reconstruction loss weight A. Left: M2T performance (RP3, KID). Right: T2M performance (RP3, Bleu4). GT: Ground Truth
large (low a+) smooths motion but affects semantic alignment, while large a (low X) sharpens token prediction but increases geometric artifacts, reflected in higher KID scores. The balanced setting a = 0.5, À = 0.5 delivers the best overall performance across both M2T (RP3 = 0.721, KID 0.216) and T2M (RP3 0.571, Bleu4 0.181).
When À is high (i.e., the reconstruction loss dominates), the model struggles to capture the overall distribution, highlighting the importance of the LM loss for maintaining semantic alignment. Con- versely, when a too high, the model predicts sharper discrete tokens but exhibits poorer geometric realism. These findings confirm that balanced loss weighting is essential for high-quality motion generation.
17

---

Published as a conference paper at ICLR 2026
Method
Rating (1-5)
Method
Rating (1-5)
A = Random
1.106
Random motion
1.375
B Our annotation pipeline
4.244
B Without filters
2.434
C Human annotation
4.673
C = Final-cleaned
4.133
Table 14: User study results. Left: annotation quality ratings. Right: motion quality ratings. Rating:
1 = Low, 5 = Best
C 3D HANDS IN THE WILD (3D-HIW) DATASET EXTENSION:
C.I DATASET ANALYSIS CONTINUATION:
To extract 3D hand motion reconstructions from egocentric videas, we first process high-level text descriptions from the Vid5M dataset to identify sequences involving human presence, particularly those where humans interact with objects. We then cluster these descriptions into scene-level categories (c.g.. crafting, repair) and sample uniformly across clusters to mitigate the overrepresentation cooking activities. We also study the impact of filter with respect motion quality in Table [3] where ablate key components of our cleaning pipeline. Removing filters (c.g.. hand visibility checks, acceleration constraints, temporal smoothing) significantly degrades R-Precision and increases motion noise. Further, we analyze the distribution of top-35 verbs and nouns in our dataset which is presented in Figure
Table 13: Dataset design choices evaluation
Method
RP3 f
MMD 4
w/o. both hands filter
0.178
3.819
w/o. acel.
0,422
2.653
w/o. temp smooth.
0.511
2.249
w/o. verifier filter.
0.553
2.019
Ours (final)
0.666
1.903
1990
2441
paper
Top 35 Nouns
Top 35 Verbs
Figure 11: Top-N Verb and Nouns: We present the distribution of top-35 verbs and nouns in the '3D Hands in the wild' (3DHiW) dataset
C.2 PERCEPTUAL USER-STUDY:
Motion Reconstruction: We conducted an additional MTurk user study to assess the perceptual quality of our reconstructed hand motions. Workers were shown the input egocentric video alongside two rendered 3D sand-motion reconstructions (front and back views), and were asked to rate on 1-5 Likert scale how realistic the 3D motion appeared and how well it matched the motion in the
18

---

Published as a conference paper at ICLR 2026
Input
"The restiin the honds the 30 andiaationn en meaitto and aloualyy yre erdain the marlis the Inguet
mochhere madee.
the oiject armatio
Touena vncr oid
"S/DAT servoled wbbert
Anlination
you ogree Nelloiing
camplnatyy cuagge
Agras
compasiry Agos
Figure 12: MTTurk interface used for the motion user study.
video. We evaluate three categories: (A) random motions sampled from unrelated sequences, (B) our reconstruction without filtering, and (C) our final filtered reconstruction. From Table users overwhelmingly preferred our final reconstruction (4.133) compared to the unfiltered version (2.434)
and the random baseline (1.375). We restricted participation experienced MTurk workers (>5000
HITs, 298% approval rate) and collected ratings on 65 sampled videos, with each video evaluated by 25 unique workers, resulting in a total of 1,625 judgments. The marked improvement from (B) to (C) confirms that our filtering pipeline substantially enhances motion quality. The MTTurk user-study interface presented in the Figure
Text annotation: In addition, conducted human evaluation of the generated annotations using an MTurr study that mirrors the setup described above. Workers were shown an input egocentric video together with candidate text description, and were asked rate 1-5 Likert scale how much they agreed with the statement: "The text accurately describes the hand motion in the input video." " We evaluate three categories: (A) a random annotation sampled from human annotation's (B) our generated annotation, and (C) the corresponding human-written annotation. As reported in Table 14, random annotations received very low scores (1.106). confirming that workers reliably detect mismatched or incorrect text. Our generated annotations achieved a high rating of 4.244, which close to the human- written descriptions (4.673). This strong alignment indicates that our automated annotation pipeline produces realistic and human-quality motion descriptions that accurately reflect the hand motions in the video. The MTurk interface used for this annotation study is shown in
Figure 13
19

---

Published as a conference paper at ICLR 2026
Text description:
The person spreads butter on bread
Figure 13: MTurk interface used for the text annotation user study.
C.3 TEXT ANNOTATION PROMPTS:
Here, give further details of the prompts introduced in Section 3.1 and Figures 3 and In order to give the reader a better understanding of what is requested in the prompts, we give simplified (i.e. natural-language-baseel
prompt summaries in Figure [4] The actual exact prompts passed into the annotating LLM contain more formal language as well as strict JSON output specification (following the example of (Shorten (2024)). The final prompts of both stages are given in Figure 15
20

---

Published as a conference paper at ICLR 2026
Stage Open-ooalllaryy lighteed sccocntion prangh
Accee- Disciipiia
Trandliss
when clirg di roins
ontere eljers served
Owe't evetice colers mewh, somen furm right
uhe lie "wokitc
poot"
jar
Ortr demils dees
desils trataterr
The eouut
Fongent
Tha cheeaaaee after.
Desember deffeeeee octter oftert poirs veree
adjact pestiimen meluter the fand alans, srupeed sna)
Cylenul short oasr prore Picreedione riie "weir Rd"" Rd's
hendrrc caod the Hhrry from
cheeg
The pedtr
somp acin
Sorinns the pai the Huunk slas Oagerr wolld lory decona fiett
All cnaniae might be
"The sa suitte turte Vrt the bot
Haaal
"The foom pedte
Your an denobbe haw the perse trviig teec Fuads
of essrrle wight weld jarr heef
right
apprrsc
wilre sepple
pronde partt
plriley
Eveys, vey
cpn
Spenes't the hands same simonetec acheer)
Aftorni the Herved
decoriptio hard rekes explie
doing sing
purmou
fur evee fete- jo rely eherg somer lang
ooy sigerr abjer imersteee with bodt jest
Saporrtig tur eriease that shoer chare parport may ldlie
Pute charee in otjec fig. donee opert
Cosamed achiss ita. "urora
Ciaacs indieddin Jer" (Wow')
Which fnger
Mfuat, matiols valrorn)
denionaa eles
moop curp wight
commok peruur
"The ber aottr the whlte rhabiizen hotto enrly"
Tenk Suedaleryy teadetker Chack fesparintes
spe-crmnic
youth danallad respatass unicciss rabl tache accaacc how rákas akit toach and))
grn cenirer tus referme vohand, loyk levi reptae
denumber pervents ling
Shom with forect Thi poner octtoon sajetharr.
lecaae interaation
by carler Sorts
hight? Surted, una ribut rntarnnss
enbiliinn hou?
hoad canon""
Candroabbinn Eangraked
Yo age-ormiic vitu ananctinn
craal amotricce for the actor decriptios hos nias rick vecetow and Insund into
lepirs
fy activer abjuce Shape mutck then inedeed
For each seros deseibe
var the qpxt aed rosso
the Art ieeeeed
ongect tig
Aegentt
epporting oolenss anahar heppors
Opreech the wés Beft or rightt cisrrry wiitee mouue fut wh shoclendy comenn
Finingga sed camnayy (6-3 aumenass
that
from enanae
reprrree
Do neer
proddss
Esagls a Enerenaled awerary.
"mepeeiii lest bultreed wh starkunn fen yeenel doreet Howiry
Figure 14: Simplified natural language prompt summaries. First stage (top): First tasks are used for PCOT, and Task 5 is the open vocabulary summarization of the output of the first 4 tasks. Second stage (bottom) is used for the final closed-vocabulary fine-grained annotation generations.
21

---

Published as a conference paper at ICLR 2026
Talk Addr Busclppia
apaeriic cl Eireet orten ciject wirt colern krtiiighn ratee dirtet col cret
well emvol peropec wine Hy Encrrt ddpt sidr ford ph, tee!
For rly, grecrete W home, for 64I1 tiee geeerev TWI STRTTIN
For geonsts TWO STEDNA:
Tpeel benee qualts coreenty Thate partt
"hur inyad pestrrty Rauters off
Secrel shngg 5TTTE EOTT with iefctted from
antect intection in Secert ceneer censter
therr gl (Tumas Lud's Plass 1447)
Dest
krom tweeh
fallorin (Sodeiarly
Satur Arnent
tne follaing foram (230%odl
reeech, (hyvet morane tytem
"slats, sowerry
"aaae "poeetpatinne pooperccooen Aud "pust sner)
somatyy I
Heall Bale
preci spaiiin vill badd aned Vo abaus duall uppardd
Toik riedin
for geerete TWO arREwe
You agioorth is Adt cacctrar
Prover crtin lntee TME ir ronrly reporellty fevale
TWO
Calcc kape ccaist
preide
that WF
srry. sabart sapude deou mumiigg the for
Snowd strng cirnorre cool
siden parn, (Taldt, Tour beil"" "pent" "Maadar"
Eahwiee Leef
beun fena:
Brtur the relleey farmel LINON only coneentty tooleg
Goappie:
(Teal (Tit alsondupeet) fiee recheeeeee Sheft, Dene
eicklly not oerratio
(y
Tank in Ope laakader teadek Clack haseainnies
Sill pooid dee aponniic vieu and ruppresa to serrt dies. Uss thei repuear geonne iis ii-in open moraloy corice
is achar actanotgee rigaer now eu extuirtunnion a riga A4 lntart ars tnd whula asmenca ducorptio the fwo The parse schaon sctan-aejut waguetarm
New noart how retoraae kctrliges
PA44, Rasibe cntracitionn ananess fuzte 1V72.
calar dost arows seacoen tuct Auli
"ihe andeessiinn
rabur
maccoslatiin hend hard resnee ceneers to show corectt. "roeb
Open woclubery afleed nor nomner
Outset
Ellen
Peper Seeee kigh Somel oe i "aagg seepiiin the parses siaoea"1)
Wag Claud venbalers the darigroust comcakke
You age soeris viea ln espart
Degee
Tadk:
exdeemni peatly lend desd with codee
jeueer capo duleaey
Shete wed des thee colen camtary (eclard coley coneccs 2100 16 (011) relatio, disice Tinter MoRICe
cn andooe oiic MCTEC Sactis and laae plise
The exact formal prompts used in the data annotation pipeline. First stage (top): First 4 tasks are used for PCOT, and Task 5 is the open vocabulary summarization of the output of the first 4 tasks. Second stage (bottom) is used for the final closed-vocabulary fine-grained annotation generations. The prompts were designed following Shorten et al. (2024).
Figure 15:
22

---

Published as a conference paper at ICLR 2026
D RELATED WORKS
In contrast to prior work based on controlled mocap datasets or single-codebook tokenizers, we contribute the first in-the-wild 3D hand motion dataset with large-scale semantic annotations, part-modality decomposed tokenizer for robust hand representation, and a geometry- aligned LLM training strategy. Together, these contributions enable CLUTCH to synthesize natural, diverse, and semantically consistent hand motions in unconstrained real-world settings.
Discussion:
D.1 MOTION DATASETS/ / ANNOTATION
Existing motion datasets provide a foundation for body-level modelling but remain limited for hands. AMASS (Mahmood et al., 2019) aggregates mocap sequences, while GRAB. ARCTIC, H2O, Dex YCB (Chao et al.) 2021). and OakInk (Zhan et al.) 2024] Yang et al. 2022) offer detailed 3D hand-object interactions. More recently, Gigahands (Fu et al. 2025) introduced large dataset of 15K hand motion sequences with diverse actions and objects. However, these datasets are costly to collect, restricted to controlled studio settings, and cover only narrow action sets. Large-scale egocentric datasets such Ego4D (Grauman et al. (022) and Vid5M (Wang et al. 2024) capture diverse real-world activities, but lack accurate 3D hand reconstructions with semantic labels. This gap has far prevented hand motion modelling from benefiting from large-scale training methods that have driven rapid advances in vision and language.
Motion Datasets:
Egocentric motion captioning: Recent advances in egocentric video understanding have leveraged natural language for supervision, moving beyond classic action recognition tasks. LaVila (Zhao al. 2023). HOD (Pei et al.) 2025), and EgoLM (Hong et al. 2024) are closest to our work on egocentric video to motion captioning. LaVa and EgoLM leverage large language models (LLMs) generate dense narrations for videos, while HOD augments these narrations by integrating detected trajectories with motion cues to produce semantically richer descriptions. In contrast, our method introduces two-stage annotation pipeline: high-level open- ppenvooabulaary reasoning via parallel chain-of-thougle prompting, followed by closed-vocabulary fine-grained grounding. This design reduces hallucinations, improves consistency, and yields scalable annotations tailored for text-to-motion modelling.
D.2 MOTION MODELLING
Full-body and Gesture Motion modelling: Research in motion generation has largely focused on
full-body and gesture synthesis (Guo et al. 2024 [ Liu et al. 2023 Zhang et al. 2023 Jiang et al.
2025| Wang et al.) 2223 [hafirr et al. 2023 Xie et al. [0033] Karunratanakul et al. 2023 Zhang et al.
2025c | Athanasiou et al.j 2024 Chi et al. 2024; Chen et al. 2024 Liu et al. [2024] Habibie
ct al. 2024). Recent models, such as MDM Tevet et al. (202)) and MotionGPT Jiang al. (2024),
architectures and large-scale motion datasets to generate realistic human movements. Further, (Chen al., 2024) built an multi-modal language models to unify the verbal and non-verbal 3D human motions. These approaches demonstrate strong performance on body-level actions but are primarily trained on controlled studio data, limiting their ability to generalize to fine-grained, unconstrained hand dynamics. While effective for large-scale gestures or locomotion, they fall short in modelling the nuanced variability of everyday hand behaviors.
leverage transformer-
3D Haaddmmtton modelling: A smaller body of work explicitly targets 3D hand motion modelling, where hands are modelled using MANO (Romero et al., 2017) and objects 3D meshes. Recent works such as HOIGPT (Huang et al. 2025), and other band-object interaction models (Christen et al. 2024. Cha al. 2024 Li et al.) 20256 Ghosh al., 2023) aim capture fine hand-object interaction. However, they rely on high-quality mocap datasets such as GRAB Taheri et al. (2020). ARCTIC (Fan) et al. 2023), and H2O (Kwon et al., (02T). , which are limited in scale and diversity. Consequently, current hand motion models are often limited to narrow distributions of scripted actions.
LLMs for motion modelling: Large language models have recently been adapted for motion generation, leveraging their strengths in sequence modelling and cross-modal alignment. Works such as (Jiang et al. 2024 Huang et al. 2025 Chen et al. 2024) treat motion tokens as text-like symbols, enabling pretrained LLMs to transfer to motion tasks. While promising, these methods are limited by small-scale datasets and training objectives that emphasize token prediction accuracy rather than reconstruction fidelity. EgoLM (Hong et al. 2024) addresses this by introducing soft-linear blending
23

---

Published as a conference paper at ICLR 2026
regression losses during pretraining, improving rext-motion alignment. However, such regression objectives confiit with cross-entropy: blending encourages smooth interpolations, whereas CE enforces sharp token choices, leading to ambiguous representations and reduced generalization. Our approach extends this line of work with geometry-alignment stage after pretraining, where Gumbel-Sofine sampling and hand motion reconstruction losses guide the L.MM toward motions that are both semantically grounded and geometrically consistent.
VQVAE motion-prior: Recent approaches discretize motion using VQ-VAE tokenizers, enabling motion to be represented in a language-like manner. Works such as (Jang et al. 2024 Zhang et al. 2023) show that modelling motion as a sequence of tokens facilitates cross mooss learning with text. However, standard single-codebook tokenizers struggle to capture the multimodal nature of motion, where both trajectories and poses of different body parts must be jointly encoded. To address this, (Yi et al. 2023) introduce compositional codebooks for hand and face motion, while (Huang al. 2025) employ separate codebooks for hand and object motion. Similarly, (Chen et 2024)
decompose body parts into individual codebooks, each modeled independently. (Wang al. 2025
further explore scaling strategies for codebooks to improve motion representation capacity. Building these ideas, formulation extends compositional quantization by introducing distinct codebooks for trajectories and hand poses, and further disentangling left and right hands during encoding and decoding. This design improves efficiency and generalization under higher temporal compression, while providing finer-grained control over multimodal hand motion generation compared to prior works.
24


Reviews:
[{"review_id": "oflSKPB11M", "reviewer": "Reviewer_x3VX", "summary": "This work presents a system for text-conditioned 3D hand motion generation and captioning in in-the-wild settings. The two primary contributions:\n1. A large-scale dataset of 32,000 3D hand-motion sequences paired with textual descriptions, sourced from egocentric videos (Ego4D). \n2. An LLM-based system that models motion tokens. A new VQ-VAE tokenizer that decomposes hand motion into separate codebooks for trajectory and pose\nThe authors demonstrate their experimental results on their new benchmark for both text-to-motion and motion-to-text tasks.", "questions": "1. How much computational resource and time does it take to generate a dataset of this size? It's key to show whether the proposed method is scalable.\n2. The paper says currently their current method doesn't support HOI generation. How is Figure 1 generated? Do you manually align the hand trajectory to fit the objects?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The 3D-HIW dataset is a good contribution. The proposed VLM-based annotation pipeline is a clever and scalable approach to captioning this in-the-wild data.\n2. The paper does a good job of validating its design choices with the ablations for the SHIFT tokenizer and the training stages.\n3. The paper is well-written, the figures are informative, and the core ideas are articulated clearly.", "weaknesses": "1. Lack of Hand-Object Interaction (HOI) is the most significant limitation. The paper frames its work as \"in-the-wild\"  yet the model only generates 3D hand motion. It does not model the objects being interacted with. True in-the-wild motion is almost entirely defined by HOI, which is explicitly left as future work.\n2. The model is trained and evaluated exclusively on the authors' new dataset. It is unclear how CLUTCH would perform on other public benchmarks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4B3gzw6F0n", "reviewer": "Reviewer_oQBC", "summary": "The researchers present two main contributions:\n1. A large-scale dataset, 3D Hands in the Wild (3D-HIW): They construct this dataset using 3D hand trackers to extract hand trajectories from egocentric videos and leverage vision-language models to generate corresponding textual annotations.\n2. The CLUTCH model: A transformer-based model trained on 3D-HIW, introducing two key innovations: (a). SHIFT: A novel tokenization method that decomposes hand motions into separate trajectory and pose components for each hand, improving generalization and yielding more accurate motion reconstructions. (b). Geometric Refinement: A fine-tuning stage applied to the language model that enhances the geometric accuracy and realism of the generated animations. This includes a reconstruction loss directly applied to the decoded 3D motion parameters.", "questions": "1. What \"Div →\" means ?\n2. The presentation could be improved, for example, inconsistent title capitalization (Line458), text size in tables, and line-breaking logic (Line271).\n3. Because text is discrete while motion is continuous, it may be more natural to model hand motions using architectures like TransFusion, similar to how VLMs such as Pi0?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The motion dataset represents a highly valuable contribution to the field.\n2. The SHIFT mechanism is well-motivated, effectively decoupling trajectory-level movements from fine-grained finger motions, and ablation studies demonstrate its effectiveness. Also, enabling bidirectional motion–text decoding is an innovative design, and it is noteworthy that this approach performs successfully in practice.", "weaknesses": "1. Since the hand motions are synthesized from a single textual description, how is motion diversity ensured? Are there mechanisms in CLUTCH to generate varied hand trajectories or poses from the same caption?\n2. While the proposed approach is effective for isolated hand motions and the datasets are centered on hand-only movements, its omission of object interactions could constrain its applicability to more realistic, object-involved settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hTPtO0pgOQ", "reviewer": "Reviewer_kcC7", "summary": "The work introduces a large-scale dataset of human hands automatically extracted from a large-scale egocentric video dataset, consisting of motion sequences and accompanying textual activity descriptions. Additionally, a method for text-to-hand trajectory generation and hand trajectory-to-text description is proposed, benefiting from a novel hand tokenization scheme. The proposed method outperforms multiple baselines.", "questions": "In Section 3.3, what is meant by \"top-200\" and \"top-3000\"?\n\nWhat is the purpose of the introduced \"testing\" split if numbers are reported on the validation split only?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The scale of the hand motion dataset is unprecedented. The work provides a thorough analysis of the introduced dataset. The proposed method outperforms multiple baselines on the introduced dataset. The design choices of the method and filtering of the dataset are quantitatively supported by ablation studies. The proposed method supports both the text-to-motion and motion-to-text tasks simultaneously.", "weaknesses": "For a work introducing a novel dataset as its main contribution, more qualitative examples of the generated trajectories as well as hand poses and coarse/fine-grained textual descriptions are necessary. This is a major weakness, especially coupled with the following concern:\nThe noun distribution in Figure 11 shows several undesirable entries being common in the dataset, e.g. \"hand\" (hand touching a hand?) and \"cut\" (a verb?). This raises questions about the quality of the dataset's noun/verb annotations.\nThe efficacy of the method is supported by its strong performance against baselines. However, at no point does the work mention any human verification of the generated dataset. As such, it is difficult for a reviewer to ascertain its quality. A human study would have greatly benefited the work. The contribution from the dataset side is thus limited for me.\n\nIn addition to insufficient qualitative examples of the dataset, more qualitative examples of the method's output must be included.\n\nThe proposed method was only evaluated on the introduced dataset, and not on other datasets such as GigaHands or ARCTIC.\n\nIt would be good to add qualitative examples of trajectories to the rightmost t-SNE plot in Fig. 6. Merely covering a broader t-SNE range of hand poses could also be achieved by a large fraction of erroneous poses in the dataset.\n\nThe object is not at all considered in the proposed dataset and method, limiting their usefulness.\n\nThe work could benefit from a table comparing it to existing datasets in terms of scenario count, total length, diversity (number of objects), etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ADq5NoIK49", "reviewer": "Reviewer_5Tyz", "summary": "This paper addresses the problem of text-conditioned hand motion generation in unconstrained \"in-the-wild\" settings. The authors make three primary contributions: (1) a novel data annotation pipeline combining VLMs and 3D hand trackers to create the 3D-HIW dataset with 32K hand motion sequences and aligned text descriptions, (2) SHIFT, a part-modality decomposed VQ-VAE tokenizer that separately encodes trajectory and pose for left and right hands, and (3) CLUTCH, an LLM-based system featuring a geometric refinement training stage that applies reconstruction losses directly to decoded motion parameters. Experiments demonstrate improvements over baselines including HumanMDM, MotionGPT, and T2M-GPT on both text-to-motion and motion-to-text tasks.", "questions": "1. Have you conducted human evaluation of the generated annotations? What is the agreement between your automated annotations and human-written descriptions?\n2. How does the model perform on motions significantly different from the training distribution? Can it generate novel compositions of actions?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The focus on in-the-wild hand motion generation addresses a significant gap in the literature, moving beyond studio-captured datasets with limited diversity.\n2. The 3D-HIW dataset with 32K sequences represents approximately 10× the scale of GRAB and ARCTIC, offering substantially greater diversity in actions (1045 verbs, 1355 objects) and scenarios.\n3. The two-stage approach using Parallel Chain-of-Thought prompting followed by closed-vocabulary refinement is well-motivated and demonstrates improved GPT-scores (6.9) compared to existing methods.", "weaknesses": "1. Dataset quality concerns:\n- The reliance on HaWor for 3D reconstruction may introduce systematic errors or artifacts that propagate through the entire pipeline.\n- The filtering criteria (80% hand visibility, acceleration thresholds) may introduce biases toward certain types of motions.\n- No human evaluation or validation of the reconstructed 3D motions is provided.\n\n2. Geometric refinement stage lacks clarity:\n- The Gumbel-Softmax formulation is mentioned but not detailed.\n- The balance between cross-entropy and reconstruction loss (α, λ) appears critical but hyperparameter sensitivity is not thoroughly analyzed.\n- The comparison to EgoLM's soft-blending approach (Table 5) shows only marginal improvements, raising questions about whether the added complexity is justified.\n3. The paper primarily compares against methods designed for full-body motion (HumanMDM) or general motion (MotionGPT). Comparisons with recent hand-specific methods or SOTA human motion generation method (MoMask) are missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)