UniHand: A Unified Model for Diverse Controlled 4D Hand Motion Modeling
Authors: Zhihao Sun
Tong Wu
Ruirui Tu
Daoguo Dong
Zuxuan Wu
Venue: ICLR 2026 Poster (2026)
Keywords: Hand Motion Modeling
Diffusion Model
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Hand motion plays a central role in human interaction, yet modeling realistic 4D hand motion (*i.e.*, 3D hand pose sequences over time) remains challenging. 
Research in this area is typically divided into two tasks: 
(1) Estimation approaches reconstruct precise motion from visual observations, but often fail under hand occlusion or absence; 
(2) Generation approaches focus on synthesizing hand poses by exploiting generative priors under multi-modal structured inputs and infilling motion from incomplete sequences.
However, this separation not only limits the effective use of heterogeneous condition signals that frequently arise in practice, but also prevents knowledge transfer between the two tasks.
We present **UniHand**, a unified diffusion-based framework that formulates both estimation and generation as conditional motion synthesis. 
UniHand integrates heterogeneous inputs by embedding structured signals into a shared latent space through a joint variational autoencoder, which aligns conditions such as MANO parameters and 2D skeletons.
Visual observations are encoded with a frozen vision backbone, while a dedicated hand perceptron extracts hand-specific cues directly from image features, removing the need for complex detection and cropping pipelines.
A latent diffusion model then synthesizes consistent motion sequences from these diverse conditions.
Extensive experiments across multiple benchmarks demonstrate that UniHand delivers robust and accurate hand motion modeling, maintaining performance under severe occlusions and temporally incomplete inputs.

Published as a conference paper at ICLR 2026
UNIHAND: A UNIFIED MODEL FOR DIVERSE CON- TROLLED 4D HAND MOTION MODELING
Zhihao Sun!.2 Tong Wu², Ruirui Tu 12, Daoguo Dong ! Zuxuan Wu
Institute of Trustworthy Embodied AI, Fudan University Shanghai Key Laboratory of Multimodal Embodied Al
Stanford University
"Correspondence Authors
ABSTRACT
Hand motion plays central role in human interaction, yet modeling realistic 4D hand motion (i.e., 3D hand pose sequences over time) remains challenging. Re- search in this area is typically divided into two tasks: (1) Estimation approaches reconstruct precise motion from visual observations, but often fail under hand oc- clusion absence; (2) Generation approaches focus synthesizing hand poses by exploiting generative priors under multi-modal structured inputs and infilling motion from incomplete sequences. However, this separation not only limits the effective use of heterogeneous condition signals that frequently arise in practice, but also prevents knowledge transfer between the two tasks. We present UniHand, a unified diffusion-basse framework that formulates both estimation and genera- tion conditional motion synthesis. UniHand integrates heterogeneous inputs by embedding structured signals into shared latent space through a joint vari- ational autoencoder, which aligns conditions such as MANO parameters and 2D skeletons. Visual observations are encoded with a frozen vision backbone, while a dedicated hand perceptron extracts hand-specific cues directly from image fea- tures, removing the need for complex detection and cropping pipelines. A latent diffusion model then synthesizes consistent motion sequences from these diverse conditions. Extensive experiments across multiple benchmarks demonstrate that UniHand delivers robust and accurate hand motion modeling. maintaining perfor- mance under severe occlusions and temporally incomplete inputs.
INTRODUCTION
The human hand plays a central role in our interactions with the world. not only allows us to manipulate tools with dexterity but also to communicate through gestures. Given this importance, modeling realistic 4D hand motion (i.e. 3D hand pose sequences over time) has emerged as an active research problem in computer vision and graphics. Progress in this field crucial for applications such as virtual reality (VR). digital avatars, and robotics (Qi et al., 2024; Xie et al., 2025; Jiang et al., 2025).
Existing research in 4D hand modeling is predominantl divided into two distinct tasks, each typ- ically addressed by specialized models. Estimation approaches aim reconstruct precise motion directly from visual observations, such as monocular or multi-view videos. These methods, however, often struggle with hand occlusions (Duran al., 2024), temporally incomplete frames (Pavlakos et al., 2024; Dong et al., 2024), and tasks requiring flexible editing. Generation approaches, on the other hand, focus on synthesizing hand poses by exploiting generative priors under multi-modal structured inputs, such 2D and 3D skeletons (Wan et al., 2017; Yang et al., 2019; Li et al., 2024), and infilling motions from incomplete sequences (Zhang et al., 2025; Yu et al., 2025).
This separation between estimation and generation not only restricts the effective use of heteroge- neous condition signals that commonly arise in real-world scenarios, but also prevents the transfer of knowledge and motion priors across the two tasks. When accurate reconstruction is required, rich visual observations, such as images or videos, are indispensable. In contrast, for motion synthesis or editing, structured conditions such as 2D skeleton keypoints and MANO parameters often more

---

Published as a conference paper at ICLR 2026
suitable due to their ease of manipulation. In practice, visual inputs may be affected by hand occlu- sions or absence, while other condition signals can exhibit temporal discontinuities. These diverse and potentially incomplete conditions underscore the need for unified framework that can flexibly integrate heterogeneous conditions and information.
Recent research has highlighted the potential synergy between estimation and generation. Some studies adopt multi-sttge frameworks that exploit generative priors to refine or complete the hand pose sequences detected by estimation methods (Zhang et al., 2025; Yu al., 2025). Other works explore unified generative approaches that support multiple modalities of input, thereby bridging the two tasks within a single formulation (Li et al., 2024). Building on these insights. we further extend this direction by exploring multimodal alignment and flexible condition integration, and introduce UniHand, unified diffusion- -aased framework for 4D hand motion modeling under heterogeneous conditions. For structured signals such as MANO parameters and 2D) skeleton keypoints, UniHand employs joint variational autoencoder to align multiple encoders within shared latent space, enabling all structured signals to be fused during the diffusion process. For visual observations, which are common and information-rich. particularly in estimation scenarios, UniHand uses a frozen vision backbone extract features from full-size frames and hand perceptron module to attend to hand-relevant tokens. A latent diffusion model then integrates multiple conditions to generate the final motion sequence. Motion is generated in a canonical camera space defined by the first frame, ensuring consistency under both static and dynamic cameras without relying on extrinsic calibration. By integrating diverse structured and visual conditions, UniHand unifies accurate estimation and flexible generation within a single framework. Our contributions can be summarized as follows:
We propose UniHand, the first unified model that formulates both 4D hand motion estimation and generation as conditional motion synthesis. Our diffusion-based model flexibly integrates heterogeneous conditions.
We design a joint variational autoencoder that aligns structured signals into shared latent space, and introduce a hand perceptron module that directly attends to hand-related features from dense tokens extracted from full-size frames.
We conduct extensive experiments multiple benchmarks and demonstrate that UniHand achieves robust and accurate motion generation, particularly under challenging scenarios such severe hand occlusions and temporally incomplete signals.
2 RELATED WORKS
2.1 HAND MOTION ESTIMATION
We first review research on hand pose estimation, where methods take visual observations as in- put to reconstruct hand pose or motion. Early works relied depth cameras reconstruct 3D hands (Ge et al., 2016; Oikonomidis et al., 2011). With the introduction of the MANO paramet- ric hand model (Romero al., 2017b), Boukhayma et al. (2019) proposed the first (earning-based approach that directly regresses MANO parameters from RGB inputs, inspiring a line of follow-up studies (Back al., 2019; Zhang al., 2019). Other works adopt a non-parametric strategy and directly predict the 3D mesh vertices of the MANO model (Kulon et al., 2019; Ge et al., 2019; Choi al., 2020; Lin et al., 2021b). Recent studies have emphasized the importance of scaling both data and model capacity. HaMeR (Pavlakos et al., 2024) investigates this direction by combining large-scale training data with large Vision Transformers (ViT), while WiLoR (Potamias al., 2025) introduces data-driven pipeline and refinement strategy for efficient multi-hand reconstruction.
While most approaches focus on image-based estimation, they can also be directly applied to videos. However, this often ignores the temporal information contained in videos and struggles with chal- lenges such as occlusions and fast motion. Deformer (Fu et al., 2023) implicitly reasons about the relationship between hand parts within the same image and across timesteps. HMP (Duran et al., 2024) exploits motion priors to enable video-based hand motion estimation through latent optimiza- tion. HaWoR (Zhang et al., 2025) reconstructs hand motion by decoupling hand pose reconstruction in camera space from camera trajectory estimation in the world frame. Dyn-HaMR (Yu et al., 2025) extends this idea with multi-stage, hand pose tracking and SLAM methods to model interacting hands under dynamic cameras. How- ever, existing methods generally rely multi-stage detection-based pipelines and cannot flexibly
optimization pipeline that relies external
2

---

Published as a conference paper at ICLR 2026
incorporate diverse types of conditions. In this work, we instead view hand pose estimation special case of conditional motion synthesis, which enables a unified hand motion generation.
2.2 HAND MOTION GENERATION
Human motion generation has been widely studied under diverse condition signals, including text (Tevet et al., 2023b; Jin et al., 2023), actions (Guo et al., 2020), speech (Alexanderson et al,, 2023), music (Tseng et al., 2023), and scene (Hassan et al., 2021; Yi al., 2024). In contrast, hand motion has not typically been conditioned on such a broad range of modalities. Most existing approaches focus on hand-object interactions (HOI), where object geometry serves as the primary prior for synthesizing plausible grasps and interaction sequences. For example, GraspDiff' (Zuo et al., 2024) leverages diffusion models to directly generate grasps conditioned on 3D object mod- els, while MGD (Cao et al., 2024) learns joint prior across heterogeneous hand-object datasets for improved generalization. Sequential extension such as Text2HO1 (Cha et al., 2024) incorporates text guidance by decomposing the task into contact and motion generation. Despite these advances, the reliance on object-specific priors and task-specific pipelines limits their applicability to broader hand motion modeling.
A more general direction explores probabilistic models to learn the distribution of feasible hand poses and motions. Unconditional priors aim to capture the distribution p(z) of plausible hand poses without external inputs. Early approaches relied on biomechanical constraints, manually defining joint degrees of freedom and rotation ranges (Yang et al., 2021; Spurr al., 2020). Later studies adopted data-driven strategies, such as applying principal component analysis (PCA) to MANO pa- rameters (Romero al., 2017a) or training variational autoencoders that map hand poses into Gaus- sian latent spaces (Zuo et al., 2023). Conditional priors instead model the distribution p(z|c) under external conditions such as RGB images, depth maps, or 2D skeletons. Typical designs employ VAEs constructed in different domains and align their latent spaces to learn feasible hand configura- tions across modalities (Wan et al., 2017; Yang et al., 2019). More advanced formulations leverage score-based models to estimate the pose distribution (Ci et al., 2023). However, these approaches remain restricted to single-condition settings and struggle with temporally incomplete condition signals. In contrast, our framework employs a diffusion-based generative model that unifies diverse signals in a shared latent space and leverages vision inputs to capture hand-related features, enabling accurate 4D hand motion modeling under multimodal conditions.
UNIFIED MODEL FOR HAND MOTION MODELING
3.1 PRELIMINARIES
Problem Definition. UniHand formulates hand motion estimation and generation within a unified framework of conditional hand motion generation. Specifically, it synthesizes a hand motion se- quence {2 { N of length N based on a set of condition signals C' and set of corresponding condition masks M. The condition set C includes: video frames Cvision E N
2D skele- ton keypoints CDD E N 3D skeleton keypoints csp E g x21 and optionally hand pose parameters G. Each condition e E C' is paired with a binary mask m RR-N where m 1 if the condition signal is available at frame i, and m = 0 otherwise. This formulation allows the model to flexibly handle varying combinations of condition signals across frames.
Hand Pose and Other Conditions Representation. The 3D hand representation x² is parame- terized by the MANO model (Romero et al., 2017b), and includes hand pose '' E RR 15x3, shape 3 IR10 along with global orientation t T and root translation I' E R For 3D hand estima- tion, hand poses are typically represented in the camera coordinate space to ensure better alignment with image features. However, for videos with dynamic camera perspectives, the hand motion se- quence becomes discontinuous due to changing coordinate systems. While the world coordinate system can alleviate this issue, it does not facilitate alignment with visual observations. To address this, we introduce a canonical coordinate system, defined as the camera space of the first frame. This decouples the hand motion from the dynamic camera, providing a consistent representation across the entire sequence, while remaining applicable to both static and dynamic camera scenar-
3

---

Published as a conference paper at ICLR 2026
Condition Encoders
Vision Backbone
Hand Perceptron
AutoReg Decoder
Transformer
Sybtios
Encoder
Anchor token
Condition Encoder
Condition Denoiser
LayerNormZero
laatent
Transformer Encoder
220/5D
Motion Encoder
Self-Attention
Motion Encoder
Cross-Attention
Transformer Encoder
Lyyee
frec
Simple
Noise
LayerNormZero
Feed-Forward
AutoReg
Lxc.
Lux
Decoder
Stage I. Latent Representation via Joint VAE
Stage 4D hand Motion Generation via Unihand
Figure 1: Overview of the UniHand framework. (I) The Joint VAE aligns motion and condition encoders within shared latent space. An autoregressive decoder iteratively reconstructs motion to preserve temporal consistency. (II) The latent diffusion model trained on this latent space, where multimodal conditions are fused, and hand-relevant vision tokens are integrated into the denoiser.
ios. Consequently, the 3D keypoint conditions transformed into the canonical space to ensure consistency, More details are provided in Appendix A.I.
Overview. We propose unified framework for conditional hand motion generation, which con- sists joint variational autoencoder (Joint VAF) and a latent diffusion model. The Joint VAE (Section 3.2) comprises multiple encoders for different modalities and shared decoder, which gether tokenize motion sequences and condition signals into shared latent representation. The latent diffusion model (Section 3.3) is defined on this latent space, where it integrates hand-relevant vision features and multiple conditions. The framework is illustrated in Figure 1.
3.2 JOINT LATENT REPRESENTATION
Variational Autoencoders (VAEs) (Kingma & Welling, 2014) compress data into latent space and have proven effective in learning compact yet expressive representations. Encoding motion in this latent space mitigates the temporal inconsistencies that often arise when training diffusion models directly on raw motion sequences (Chen et al., 2023; Zhao et al., 2025). We propose Joint VAE that encodes both motion sequences and diverse condition signals into shared latent space. This alignment between MANO- based motion, 2D skeleton keypoints, and 3D skeleton keypoints encourages the latent representation to capture motion semantics that generalize across modalities. The shared space further facilitates flexible condition fusion during controllable generation.
As shown in Algorithm we design a joint encoder architecture that incorporates both a motion en- coder and multiple condition encoders. The motion encoder &m encodes the sequence {x* Ji=1 into set of latent motion tokens 2 { where each token 2 represents the hand pose of a single frame in d-dimensional latent space. In addition, a global motion token g E R2 is in- troduced to capture sequence-level information. We introduce learnable distribution tokens T, To, and the encoder predicts Gaussian parameters (Mg. from which g is sampled. This latent variable regularized via KI. divergence loss. Similarly, each condition encoder & tokenize condition signal E C' into a sequence of condition latent tokens the shared latent space and can be fused during generation. The decoder D reconstructs the motion sequence æ in autoregressive manner. At each autoregression step. it predicts a motion segment giitn conditioned on the latent tokens 2 zii+ the global token g. and an anchor token a' represent- ing the initial state of the segment. The global token provides high-level structural context, while the frame-wise wise latent tokens preserve ime-grained motion details and condition alignment. The training objective is provided in Appendix A.2.
Ec(c) E IR N which are aligned in
4

---

Published as a conference paper at ICLR 2026
Algorithm Latent representation with Joint Variational Autoencoder
Input:
hand motion r = {-²} structured conditions {c' JE= motion encoder Em, learnable distribution tokens TN and To condition encoders & and autoregressive decoder D.
Output: motion latent tokens
{= j;=1 motion global token g. condition latent tokens de
{ and reconstructed hand motion it. (z, (2. g)) < Em(2, Tu, To)
encode hand motion to latent representation > sample motion global token
2: g Nigg. 0g)
3: for e do
4:
Zc. <- (c)
b encode each structured condition to latent representation
5: end for
6: & + 0. a Linear(x*)
initialize reconstructed motion and anchor token
7: for in {-, }e) do
8:
for i = 1 to N by step size do
autoregressive rollouts
8z:i+ + D(a 2 2 .)) CONCAT g..ttt) 11: a at+ + Linear (ameer((+*n-a (
9:
autoregressive decoding with anchor and global token
D. update anchor token
end for end for
13:
return 2, 9. Zc.
3.3 DIFFUSION-BASEE BASED MOTION GENERATION
We perform diffusion- based generation in the latent space learned by the Joint VAE. Diffusion mod- els (Ho et al., 2020) define a stochastic process that iteratively adds Gaussian noise to a clean latent representation until it becomes pure Gaussian noise, and then learns reverse the process for gen- eration. Given a hand motion sequence x and its latent representation 2g E RR obtained by the encoder E, The forward process progressively transforms 2n into Gaussian noise 2/ ~ WN I) through a Markov chain: (=z zz-1) (VVV- Br 1 BeT), where {3e} } is predefined noise schedule. The denoiser model Gg learns the reverse process, which aims to transform noise back into clean motion latents conditioned on C: (0 24, C)- = (2,
Eed), where denotes the available conditions, such as vision frames and 2D skeleton keypoints, and D; is determined by the noise schedule. Following prior work in human motion generation (Shafir et al., 2023; Tevet et al., 2023a; Zhao et al., 2025), which show that predicting the clean sample yields more tempo- rally coherent motions than predicting noise, we design the denoiser Go to predict the clean latent Eo Go(z,, l, C). The predicted in is then used to compute the mean of the reverse distribution:
V 0a
20 Var(1 ā(-1)
Et.
with a, 1 Be and ā₂ IT as Following Yang et al. (2024), we incorporate the diffusion timestep / into the modulation module of an adaptive LayerNorm.
Attending to Hand-relevant Vision Tokens. Visual observations, such as images and videos, are the most common inputs hand pose estimation and provide the richest information among all modalities. They capture not only hand pose but also contextual cues from the surrounding envi- ronment and interacting objects. However, existing approaches often crop around the hand region, which sacrifices contextual information and, in the case of video, disrupts temporal consistency since camera coordinates of the cropped regions differ across time. We instead leverage a pretrained vision backbone Evision to process a full image or video frame exssion and project it into dense tokens o' E R xxxx To extract hand-relevant information from these dense features, we introduce hand perceptron module that selectively attends to hand-related vision tokens while retaining contextual cues from the environment and interacting objects. Specifically, we employ set of trainable hand tokens H = { along with an initialization hand pose token a!. as queries. The dense vision tokens
keys and values. We adopt Rotary Positional Encoding (RoPE) (Su et al., 2024) in 3D formation, following prior work (Kong et al., 2024; Yang et al., 2024), and compute the rotary frequency matrices separately for the temporal N, height h, and width dimensions of the vision
5

---

Published as a conference paper at ICLR 2026
Table Quantitative comparison of SoTA hand pose and motion modeling methods the Dex YCB test set in the camera coordinate space. Results are reported in terms of MPJPE (mm) and AUC, with statistics across different occlusion levels.
All
Occlusion (25-550%
Occlusion (50%-75%)
Occlusion (75%-100%)
Method
PA MEIPPE J
A
PA-MPIPE
AUC,1
PA-MPIPE
AUC,1
PA MPIPE
AUC,1
Spurr et (2020)
6,83
0.864
10.65
MeshGraphormer
6.41
0.872
6.85
0.863
7.22
0.856
7.76
0.845
SemiHandObi
6,33
0.874
6,70
0.866
7.17
0.857
8.96
0.821
HandOccNet
5,80
0.884
7.37
0.853
WiL.R
5.01
0.900
5.42
0.892
5.68
0.887
HANNDVV
7.27
0.855
7.74
0.845
7.71
0.846
7.87
0.843
VIBE
6.43
0.871
6.72
0.865
6.84
0.864
7.066
0.858
TCMR
6,28
0,875
6,56
0.869
0.868
6.95
0.861
Deformer
5.22
0.896
5.71
0.886
5.70
0.886
6.34
0.873
Ha WoR
4.76
0.905
5.03
0.899
5,01
0.899
UniHand
4.08
0,918
4.22
0.913
4.25
0.912
4.26
0.912
tokens. The attention mechanism is then applied as:
Attention Q, K, V) Softmax (orinaxQQ//"// /V<)/Vd))V dx Q RoPEE
(I, K RoPE(LayerNorm(H) (W PaD)),
(3)
V LayerNorm(Wvv(r)
The trainable hand tokens aggregate vision information associated with the target hand in each frame, while the initialization pose token anchors the attention process to the correct hand instance when multiple hands are present, thereby ensuring consistent one-to-one binding across the quence. As result, the hand perceptron produces single hand token h for each frame.
Integrating Multiple Conditions. Our framework supports multiple forms of conditions, which can be grouped into structured conditions and visual observations. The first group includes signals such MANO parameters, 2D keypoints, and 3D keypoints. These representations are encoded into the shared latent space by the Joint VAE and can therefore be directly fused with the noisy motion latent during denoising. The second group consists of visual inputs, from which we extract one representative hand token per frame. Rather than being fused at the latent level, these tokens are incorporated into the denoising network through attention layers every denoising step, allowing the model to attend to vision information throughout the generation process.
We adopt a two-stage training strategy, where the Joint VAE and the diffusion model are trained sep- arately, with details provided in Appendix B.2. To further enhance generation quality and condition flexibility, we adopt classifier-free guidance (CFG) (Ho & Salimans, 2022) with trainable uncondi- tional tokens. CFG is typically expressed as Go = G0(2,, t, cg) + wGG t, c,) - g0 t, c ce)), where denotes the denoising network, 2, is the noisy latent at timestep and is the CFG scale controlling the strength condition. However, motion latents do not possess natural uncon- ditional forms Cg. To address this, introduce independent learnable unconditional tokens for motion and condition representations, which match the feature dimensions of 2 and Ze, respectively. During training, a condition latent 2 is randomly replaced with its unconditional form &cp with predefined probability p. This mechanism ensures that UniHand remains robust under diverse and potentially incomplete conditioning scenarios, while also allowing fine-grained adjustment con- ditional influence during motion synthesis. Further details training and inference are provided in the Appendix A.3.
EXPERIMENTS
4.1 EXPERIMENTAL. SETUP
Datasets. To evaluate the performance UniHand under egocentric views with dynamic cameras and to compare it with existing methods, the Dex dataset (Chao et al., 2021), which

---

Published as a conference paper at ICLR 2026
Quantitative comparison of base- line hand pose estimation methods on the HO3D dataset in the camera coordinate space. Results are reported in terms of MPJPE (mm), AUC scores, and F-scores.
Table 2:
Table Quantitative evaluation of SoTA meth- ods on the HOT3D dataset in the world coordinate space. Results are reported in terms of MPJPE (mmm under different alignment strategies and ac- celeration error.
Method
PAIPPIE
AUC,t
F
F6
Method
PA-MPIPE
G-MPIPE
GA-MPJPE
AccEr
HandOceNet
9.1
0.819
0.564
0.963
HaMcR-SLAM
9.20
161.31
43.85
15.53
AMVUR
8.3
0.835
0.608
0.965
WiLoR-SIAM
10.39
HaMeR
7.7
0.846
0.635
0.980
SLAM
10.68
128.56
38.25
5.41
WiLoR
7,5
0.851
0.646
0.983
Dyn-HaMR
8.92
$9.04
23.57
5.16
Deformer
0.546
0.963
HaWoR
5.47
47.35
18.14
5.88
Ours
6,7
0,866
0.671
0.988
Ours
4.76
63.97
25.24
4.93
contains multi-view videos with hand pose annotations in the camera coordinate system. The degree of occlusion can computed, enabling analysis of pose estimation under different occlusion levels. We further report results on HO3D (Hampali et al., 2020) to assess the generalization ability of UniHand. Following Zhang et al. (2025); Yu et al. (2025), we also use HOT3D (Banerjee et al., 2025), which provides hand poses in the world coordinate system along with camera extrinsics, to evaluate estimation performance under egocentric views with dynamic cameras.
Metrics. We report Procrustes-Aligned Mean Per-Joint Position Error (PA-MPJPE) and the area under the curve of correctly localized keypoints (AUC,) to evaluate hand pose in the camera coor- dinate space. Following Hampali et al. (2020), we also include the fraction of poses with less than 5mmm and 15mm dinate space, we report G-MPJPE and GA-MPJPE following Ye et al. (2023), where alignment with ground truth performed using the first two frames or the entire motion. In addition, we compute the acceleration error (AccEr) to assess the temporal smoothness of the generated motion.
(F@5, F@ 15) computed by the official evaluation scripts. In the world coor-
4.2 HAND MOTION CAMERA COORDINATE SPACE
Hand pose estimation in the camera coordinate space provides the most direct way to evaluate the quality of motion generation conditioned on visual observations. Moreover, evaluation under chal- lenging conditions such as occlusions and missing temporal frames is particularly important, as these phenomena frequently occur in real-world videos. Following prior work (Fu et al., 2023; Zhang et al., 2025), we evaluate our method on DexYCB, dataset that provides frame-level occlusion- related annotations. We partition the test set into multiple occlusion levels. For our approach and other video-based methods, fair comparison with image-based methods.
videos as input and then compute frame-level metries, ensuring
As shown in Table 1, compare UniHand against wide range of image-based and video- based baselines across different occlusion categories. Image-based approaches include Mesh- Graphormer (Lin et al., 2021a), SemiHandObj (Liu et al., 2021), HandOccNet (Park et al., 2022), and WiLoR (Potamias et al., 2025), which process images independently and are typically sensitive occlusion. In contrast, video-based methods such S² (Tu al., 2023), VIBE (Ko- cabas et al., 2020), TCMR (Choi et al., 2021), Deformer (Fu et al., 2023), and Ha HaWoR (Zhang et al., 2025) leverage temporal context for motion reasoning and are therefore less affected by occlusion. UniHand achieves a PA-MPJPE of 4.08 and an AUC of 0.918, outperforming all image-based and video-based baselines. Even under the most severe occlusion level, our method maintains superior performance with PA-MPJPE of 4.26 and AUC of 0.912. These results highlight not only the ben- cfit of temporal modeling but also the advantages of our generative priors and the hand perceptron module in effectively exploiting visual input.
To further evaluate generalization, we evaluate our model on the HO3D dataset, which contains diverse object interaction scenarios and severe occlusions not present in the training data. As shown in Table 2. despite the domain shift, our model achieves competitive performance, demonstrating robustness to dut-o--distiibution inputs.
7

---

Published as a conference paper at ICLR 2026
Exocentric View with Static Camera DexYCB
Dyn-HaMR
UniHand (Ours)
Groundtruth
Egocentric View with Dynamic Camera HOT3D
Dyn-HaMR
UniHand (Ours)
Groundtruth
Figure 2: Visualization of generated hand poses and trajectories. The first example shows static camera scenario where the subject picks up red bowl, with significant hand occlusion. The second example recorded with a dynamic camera, where the subject picks up and manipulates a magic cube, involving large hand movements. UniHand produces more accurate hand motion by modeling motions in a canonical coordinate space, even without relying on explicit camera extrinsics.
4.3 HAND MOTION IN WORLD COORDINATE SPACE
To evaluate the global consistency of reconstructed hand motions, we conduct experiments in the world coordinate system using the HOT3D dataset, which provides egocentric videos. We consider two categories of methods: camera-space approaches, which estimate hand poses the camera coordinate system and then transform predictions into the world frame using estimated camera poses from DROID-SLAM (Teed & Deng, 2021), I), and video-based methods, which jointly infer hand and camera motion in the world space through temporal models.
As shown in Table 3, UniHand consistently outperforms both camera-space and world-space base- lines in PA-MPJPE: demonstrating the accuracy of the reconstructed hand poses. Notably, UniHand achieves the lowest G-MPJPE and GA-MPJPE among all camera-space reconstruction methods, despite leveraging explicit camera trajectories estimation for world-space conversion. Our method relies solely on visual observations to model motions in the canonical space. It achieves perfor- mance comparable to world-space methods, such as HaWoR and Dyn-HaMR (Yu al., 2025), that explicitly utilize camera parameters. In addition, UniHand obtains lower acceleration error (AccEr), confirming the temporal smoothness of the reconstructed hand trajectories in the world frame.
We further visualize the generated 3D hand motions in Figure 2. Compared to Dyn-HaMR, UniHand recovers more stable and accurate hand motion sequences, particularly under occlusions or large hand movements. Unlike baseline methods that rely on external SLAM or require per-sequence optimization, UniHand provides a unified and efficient solution for world-space hand motion gener- ation without explicit camera estimation.
4.4 ABLATION STUDY
To analyze the effectiveness of the core components and different condition signals, we conduct ablation studies on the Dex dataset under the camera coordinate setting and the HOT3D dataset under the world coordinate setting. We also report results the most challenging occlusion level (75%-100%) of YCB, The evaluation metrics follow the same protocol as described in previous experiments.
8

---

Published as a conference paper at ICLR 2026
Table 4: Ablation studies the components, design choices, and different condition config- urations during inference, evaluated on the DexYCB and HOT3D datasets. Results are reported in terms of MPJPE (mm) under different alignment strategies and AUC scores.
DexYCB-All
DexYCB-Orclaion
HOT3D
Setups
PA-MPJPE
AUC
PA-MPJPE
AUC
PA-MPIPE
G-MPJPE
GA-MPJPE
w/o. Condition Encoders
5.21
0.895
5.56
0.889
5.92
75.49
31.03
w/n. Pretrained Eviioo
6.52
0.869
6.71
0.863
8,73
146.08
39.53
w/o. Hand Perceptron
7.81
0.843
8.75
0.824
12.46
180.59
48.93
w/o. 3D RoPE
4.65
0.906
4.76
0,904
4,95
69.20
28.94
W.
4.24
0.915
4.27
0.915
4,52
53.49
23.28
W. CDD
4.75
0.905
5.43
0.891
6.37
98.17
40.42
C3D
3,99
0.920
4.17
0.916
4.15
44.61
20.73
W. and e3p
3.48
0.931
3.67
0.926
3.82
48.11
21.36
Ours (w. Cybien and (ap)
4.08
0.918
4.26
0.912
4.76
63.97
25.24
Component Ablation. The upper part of Table 4 summarizes the ablation results of different components and design choices within the UniHand framework. Setup w/o. Condition Encoder E, replaces the condition encoders in Joint VAE with an MLP that directly maps condition signals (e.g., 2D keypoints) to the latent dimension. The performance drop indicates that the Joint VAE critical for learning consistent representations, thereby enabling more effective condition fusion. Setup w/o. Pretrained Evision uses an identical vision backbone without pretraining. The performance degradation highlights the importance of pretrained visual representations in providing reliable cues for the hand perceptron module. Furthermore, both replacing the hand perceptron module with average pooling over dense vision tokens and replacing 3D RoPE with standard ID RoPE lead to clear performance decrease.
Condition Modality Ablation. We further evaluate the contribution of each condition modality by testing different inference configurations. As shown in the lower part of Table 4, using only 2D keypoints yields acceptable performance under normal conditions, demonstrating the effectiveness of latent space alignment in the Joint VAE. However, such structural information cannot be reliably extracted under severe occlusions, resulting in poor robustness. Its performance on HOT3D is also limited, indicating that 2D keypoints alone insufficient for modeling hand motion under dynamic camera movements. Using only Cvision achieves better PA-MPJPE, but its lack of explicit spatial con- straints leads weaker performance in G-MPJPE. The combination of Cvision and cap achieves the best overall performance. showing the complementarity between visual evidence and 3D structural cues. However, since 3D keypoints are not directly accessible in real-world scenarios and are mainly applicable to editing tasks, we adopt the Cvision and CaD configuration for most of our experiments. In practice, 2D keypoints can be easily obtained using pretrained detection backbones, making this setting both effective and practical.
CONCLUSIONS AND LIMITATIONS
In this work, we introduced UniHand, unified diffusion-based framework that formulates both hand motion estimation and generation as conditional motion synthesis. UniHand employs a joint varia- tional autoencoder that aligns structured signals such MANO parameters and 2D skeletons into shared latent space, ensuring consistency across modalities. In parallel, a hand perceptron module attends to hand-related features extracted from dense tokens full-size vision inputs, enabling the model to directly exploit rich visual observations without relying on hand detection cropping. Building on these components, our diffusion-based framework flexibly integrates heterogeneous conditions to generate coherent 4D hand motions. Extensive experiments across multiple bench- marks demonstrate that UniHand achieves robust and accurate hand motion modeling, maintaining strong performance under severe occlusions and temporally incomplete signals. These results high- light the effectiveness unifying estimation and generation within a single framework, and provide research directions for more general multimodal hand motion modeling real-world applications.

---

Published as a conference paper at ICLR 2026
Limitations. UniHand models 4D hand motion directly in the canonical coordinate space with- out relying on explicit camera extrinsics, thereby providing a unified treatment of both static and dynamic camera scenarios. However, under large camera movements, visual observations other structured signals alone are insufficient to ensure globally consistent trajectories. 'his limitation is reflected in evaluation: while UniHand achieves accurate pose generation and outperforms methods restricted to the camera coordinate space, its global alignment scores remain lower than optimization-base basod approaches that explicitly leverage camera extrinsics. Future work could in- corporate camera estimation into the framework, enabling more accurate trajectory reconstruction under dynamic camera settings.
ACKNOWLEDGEMENT
This work supported by the National Natural Science Foundation of China (No. 62472098) and the Science and Technology Commission of Shanghai Municipality (No. 25511106100 and No. 25511104301)
REFERENCES
Simon Alexanderson, Rajmund Nagy, Jonas Beskow, and Gustav Eje Henter. Listen, denoise, action! audio-driven motion synthesis with diffusion models. ACM Transactions on Graphics (TOG), 2023. 3
Seungryul Back, Kwang In Kim, and Tae-Kyun Kim. Pushing the envelope for rgb-based dense 3d hand pose estimation via neural rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019. 2
Prithviraj Banerjee, Sindi Shkodrani, Pierre Moulon, Shreyas Hampali, Fan Zhang, Jade Fountain, Edward Miller, Selen Basol, Richard Newcombe, Robert Wang, al. Introducing hot3d: An egocentric dataset for 3d hand and object tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025. 17, 18
Adnane Boukhayma, Rodrigo de Bem, and Philip HS Torr. 3d hand shape and pose from images in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, 2019.
Jinkun Cao, Jingyuan Liu, Kris Kitani, and Yi Zhou. Multi-modal diffusion for hand-object grasp generation. arXiv preprint arXiv:2409.0 04560, 2024.
Junuk Cha, Jihyeon Kim, Jae Shin Yoon, and Seungryul Baek. Text2hoi: Text-guided 3d motion generation for hand-object interaction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
Yu-We Wei Chao, Wei Yang, Yu Xiang, Pavlo Molchanov, Ankur Handa, Jonathan Tremblay, Yashraj S. Narang, Karl Van Wyk, Umar lqbal, and Stan Birchfield. Dexycb: A benchmark for capturing hand grasping of objects. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2021. 6, 17
Xin Chen, Biao Jiang, Wen Liu, Zilong Huang, Bin Fu, Tao Chen, and Gang Yu. Executing your commands via motion diffusion in latent space. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2023. 4
Hongsuk Choi, Gyeongsik Moon, and Kyoung Mu Lee. Pose2mesh: Graph convolutional network for 3d human pose and mesh recovery from 2d human pose. In European Conference on Com- puter Vision, 2020. 2
Hongsuk Choi, Gyeongsik Moon, Ju Yong Chang, and Kyoung Mu Lee. Beyond static features for temporally consistent 3d human pose and shape from video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
Hai Ci, Mingdong Wu, Wentao Zhu, Xiaoxuan Ma, Hao Dong, Fangwei Zhong, and Yizhou Wang. Gfpose: Learning 3d human pose prior with gradient fields. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2023.
10

---

Published as a conference paper at ICLR 2026
Haoye Dong, Aviral Chharia, Wenbo Gou, Francisco Vicente Carrasco, and Fernando D De la Torre. Hamba: Single-view 3d hand reconstruction with graph-guided bi-scanning mamba. Advances in Neural Information Processing Systems, 2024.
Enes Duran, Muhammed Kocabas, Vasileios Choutas, Zicong Fan, and Michael Black. Hmp: Hand motion priors for pose and shape estimation from video. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2024. 1,2
Qichen Fu, Xingyu Liu, Ran Xu, Juan Carlos Niebles, and Kris M Kitani. Deformer: Dynamic fusion transformer for robust hand pose estimation, In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2023. 2,
Liuhao Ge, Hui Liang, Junsong Yuan, and Daniel Thalmann. Robust 3d hand pose estimation in single depth images: from single-view to multi-view cnns. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2016. 2
Liuhao Ge, Zhou Ren, Yuncheng Li, Zehao Xue, Yingying Wang, Jianfei Cai, and Junsong Yuan. 3d hand shape and pose estimation from a single rgb image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019. 2
Ross Girshick. Fast r-cnn. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2015, 15
Chuan Guo, Xinxin Zuo, Sen Wang, Shihao Zou, Qingyao Sun, Annan Deng. Minglun Gong, and Li Cheng. Action2motion: Conditioned generation of 3d human motions. In ACM international Conference on Multimedia, 2020. 3
Shreyas Hampali, Mahdi Rad, Markus Oberweger, and Vincent Lepetit. Honnotate: A method for 3d annotation of hand and object poses. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020. 7,
Mohamed Hassan, Duygu Ceylan, Ruben Villegas, Jun Saito, Jimei Yang, Yi Zhou, and Michael J Black. Stochastic scene-aware motion prediction. In Proceedings of the IEEE/CVF International Conference Computer Vision, 2021.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 2020. 5, 18
Haoran Jiang, Jin Chen, Qingwen Bu, Li Chen, Modi Shi, Yanjie Zhang, Delong Li, Chuanzhe Suo, Chuang Wang, Zhihui Peng, and Hongyang Li. Wholebodyvla: Towards unified latent vla for 2025.
whole-body mac-maanpulltioon control. arXiv preprint arXiv:2512.
Peng Jin, Yang Wu, Yanbo Fan, Zhongqian Sun, Wei Yang, and Li Yuan. Act as you wish: Fine- grained control of motion diffusion model with hierarchical semantic graphs. Advances Neural Information Processing Systems, 2023. 3
Diederik P. Kingma and Max Welling.
Auto-encoding variational bayes.
arXiv preprint
arXiv:1312.611e 2013. 15
Diederik Kingma and Max Welling. Auto-encoding variational bayes. International Conference on Learning Representations, 2014. 4
Muhammed Kocabas, Nikos Athanasiou, and Michael J Black. Vibe: Video inference for human body pose and shape estimation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2020.
Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXiv:2412.03600 2024. 5, 16

---

Published as a conference paper at ICLR 2026
Dominik Kulon, Haoyang Wang, Riza Alp Güler, Michael Bronstein, and Stefanos Zafeiriou. Single image 3d hand reconstruction with mesh convolutions. Procedings of the British Machine Vision Conference, 2019. 2
Mengcheng Li, Hongwen Zhang, Yuxiang Zhang, Ruizhi Shao, Tao Yu, and Yebin Liu. Hhmr: Holistic hand mesh recovery by enhancing the multimodal controllability of graph diffusion mod- els. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2024. 1,2
Kevin Lin, Lijuan Wang, and Zicheng Liu. Mesh graphormer. In Proceedings of the IEEE/CVF International Conference Computer Vision, 2021a.
Kevin Lin, Lijuan Wang, and Zicheng Liu. End-to-end human pose and mesh reconstruction with transformers. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2021b. 2
Shaowei Liu, Hanwen Jiang, Jiarui Xu, Sifei Liu, and Xiaolong Wang. Semi-supervised 3d hand- object poses estimation with interactions in time. In Proceedings of the IEEÉ/CVF Conference on Computer Vision and Pattern Recognition, 2021. 7
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. 18
lason Oikonomidis, Nikolaos Kyriazis, and Antonis Argyros. Efficient model-based 3d tracking of hand articulations using kinect. In Procedings of the British Machine Vision Conference, 2011. 2
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.0777 00700999..99,,,.. 2023. 16
JoonKyu Park, Yeonguk Oh, Gyeongsik Moon, Hongsuk Choi, and Kyoung Mu Lee. Handoccnet: Occlusion-robust 3d hand mesh estimation network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022. 7
Georgios Pavlakos, Dandan Shan, Ilija Radosavovic, Angjoo Kanazawa, David Fouhey, and Jitendra Malik. Reconstructing hands in 3d with transformers. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, 2024. 1, 18
Rolandos Alexandros Potamias, Jinglei Zhang, Jiankang Deng, and Stefanos Zafeiriou. Wilor: End- to-end 3d hand localization and reconstruction in-the-wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025. 2,7
Jing Qi, Li Ma, Zhenchao Cui, and Yushu Yu. Computer vision-based hand gesture recognition for human-robot interaction: review. Complex & Intelligent Systems, 2024. 1
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System opti- mizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD international conference knowledge discovery & data mining, 2020. 18
Javier Romero, Dimitrios Tzionas, and Michael J Black. Embodied hands: Modeling and capturing hands and bodies together. SIGGRAPH Asia, 2017a. 3
Javier Romero, Dimitrios Tzionas, and Michael J. Black. Embodied hands: Modeling and capturing hands and bodies together. ACM Transactions on Graphics (TOG), 2017b. 2, 3, 15
Yonatan Shafir, Guy Tevet, Roy Kapon, and Amit H Bermano. Human motion diffusion as gener- ative prior. arXiv preprint arXiv:2303.014: 2023. 5
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.025:0 2020. 18
Adrian Spurr, Umar lqbal, Pavlo Molchanov, Otmar Hilliges, and Jan Kautz. Weakly supervised 3d hand pose estimation via biomechanical constraints. In European Conference on Computer Vision, 2020. 3,6
12

---

Published as a conference paper at ICLR 2026
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding. Neurocomputing, 2024. 5, 16
Zachary Teed and Jía Deng. Droid-slam: Deep visual slam for monocular, stereo, and rgb-d cameras. Advances in Neural Information Processing Systems, 2021. 8
Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen-Or, and Amit Bermano. Human motion diffusion model. International Conference on Learning Representations, 2023a.
Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen-Or, and Amit H Bermano. Mdm: Human motion diffusion model. International Conference on Learning Representations, 2023b.
Jonathan Tseng, Rodrigo Castellon, and Karen Liu. Edge: Editable dance generation from music. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
Zhigang Tu, Zhisheng Huang, Yujin Chen, Di Kang, Linchao Bao, Bisheng Yang, and Junsong Yuan. Consistent 3d hand reconstruction in video via self-supervised learning. IEEE Transactions on Patlern Analysis and Machine Intelligence, 2023. 7
Chengde Wan, Thomas Probst, Lue Van Gool, and Angela Yao. Crossing nets: Combining gans and vaes with shared latent space for hand pose estimation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, 2017. 1,3
Yu Xiang, Tanner Schmidt, Venkatraman Narayanan, and Dieter Fox. Posecnn: A convolutional neural network for 6d object pose estimation in cluttered scenes. In Robotics: Science and Sys-
tems, 2018. 17
Sicheng Xie, Haidong Cao, Zejia Weng, Zhen Xing, Haoran Chen, Shiwei Shen, Jíaqi Leng, Zuxuan Wu, and Yu-Gang Jiang. Human2robot: Learning robot actions from paired human-robot videos. In Proceedings of the AAAI Conference on Artificial Intelligence, 2025.
Linlin Yang, Shile Li, Dongheui Lee, and Angela Yao. Aligning latent spaces for 3d hand pose estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019. 1,3
Lixin Yang, Xinyu Zhan, Kailin Li, Wenqiang Xu, Jiefeng Li, and Cewu Lu. Cpf: Learning a contact potential field to model the hand-object interaction. In Proceedings of the IEEF/CVF International Conference on Computer Vision, 2021.
Zhuoyi Yang. Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models 2024. 16
with expert transformer. arXiv preprint arXiv:2408.0610
Vickie Ye, Georgios Pavlakos, Jitendra Malik, and Angjoo Kanazawa. Decoupling human and camera motion from videos in the wild. Proceedings of the IEEE/VVF Conference on Computer Vision and Pattern Recognition, 2023. 7
Hongwei Yi, Justus Thies, Michael J Black, Xue Bin Peng, and Davis Rempe. Generating human interaction motions in scenes with text control. In European Conference Computer Vision, 2024.7 3
Zhengdi Yu, Stefanos Zafeiriou, and Tolga Birdal. Dyn-hamr: Recovering 4d interacting hand motion from a dynamic camera. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025. 1, 2, 8
Jinglei Zhang, Jiankang Deng, Chao Ma, and Rolandos Alexandros Potamias. Hawor: World-space hand motion reconstruction from egocentric vidcos. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2025. 2,
13

---

Published as a conference paper at ICLR 2026
Xiong Zhang. Qiang Li. Hong Mo, Wenbo Zhang, and Wen Zheng. End-to-end hand mesh recovery from a monocular rgb image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019. 2
Kaifeng Zhao, Gen Li, and Siyu Tang. Dartcontrol: diffusion-based autoregressive motion model for real-time text-driven motion control. International Conference on Learning Representations, 2025. 4,5 5
Binghui Zuo, Zimeng Zhao, Wenqian Sun, Wei Xie, Zhou Xue, and Yangang Wang. Reconstructing interacting hands with interaction prior from monocular images. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023.
Binghui Zuo, Zimeng Zhao, Wenqian Sun, Xiaohan Yuan, Zhipeng Yu, and Yangang Wang. Graspdiff: Grasping generation for hand-object interaction with multimodal guided diffusion. JEEE Transactions on Visualization and Computer Graphics, 2024.
14

---

Published as a conference paper at ICLR 2026
A METHOD
A.1 HAND POSE AND CONDITIONS REPRESENTATION
Canonical Coordinate Space. We model 4D hand motion in canonical coordinate system, de- fined as the camera space of the first frame. This formulation decouples hand motion from dynamic camera movement, providing a consistent representation across the entire sequence, while remaining applicable to both static and dynamic camera scenarios. In the case of static cameras, the canonical space identical to the camera space. For dynamic cameras, the cumera-to-canonica transforma- tion is computed as:
cam-+cano Ramm cano cam->canol
T wordd x T world-ream>
Ti cam- world maps the hand pose from the i-th frame camera space to the world space, and world maps it back to the camera space of the first frame, which serves as the canonical space.
where TV
Representation. 4D hand motion sequence is denoted as {=' of length N. Each 3D hand pose x' is parameterized by the MANO model (Romero et al., 2017b), including hand pose parameters @' R11633, shape parameters 3 E RR,0, global orientation क' IR3 and root translation r E R3, The complete pose x² is therefore represented in the canonical coordinate space as: x² = (0', 1',
The 3D skeleton keypoint condition is obtained by regressing joints from the MANO parameters using the MANO joint regressor J. All joints are transformed into the canonical coordinate space to ensure temporal consistency across the sequence. The 2D skeleton keypoint condition is derived from the projected 3D joints. We preserve the projection defined by the first-frame camera and normalize the coordinates into the range [0, 1] according to the frame resolution, which serves consistent visual reference throughout the sequence.
A.2 JOINT VAE
Architecture. Our Joint VAE adopts transformer-based architecture. Both the motion encoder E condition encoders Ec, and the decoder D are composed of 9 transformer encoder layers. Each layer configured with dropout rate of 0.1, a feed-forward dimension of 2048, hidden dimension of 512, 8 attention heads, and the GELU activation function. The latent space is defined with dimension of 512. The autoregressive decoder processes motion in segments of length We apply Rotary Positional Encoding (RoPE) as temporal positional encoding for the hidden states.
time.
Losses.
The Joint VAE trained with a composed loss defined as:
CJoincVAE = Liee + CKL + Clatent + Waux Caux-
(5)
The reconstruction loss free encourages the reconstructed motion sequence to match the ground- truth motion sequence æ. It consists of two parts, the MANO parameter reconstruction loss mannoouc and the joint reconstruction loss Ljoinuresi
Lre = Lmanourec + Wyoint Sjointree.
(6)
The MANO parameter reconstruction loss directly penalizes differences between predicted and ground-truth MANO parameters:
Emanouree (.)
where Fi, denotes the smoothed LI loss (Girshick, 2015). The MANO joint reconstruction loss penalizes discrepancies between the 3D joints regressed from the predicted and ground-truth MANO parameters:
Cyoint ree (TT))
where J denotes the MANO joint regressor.
The Kullback-Leible divergence regularization term LxT. (Kingma & Welling, 2013) regularizes the latent space learned by the Joint VAE by penalizing the divergence between the predicted latent distribution q(= H) and standard Gaussian AO, I) as:
LK.. KT(((g) [ ()][ I)),
15

---

Published as a conference paper at ICLR 2026
where the KLL denotes the Kullback-Leibler (KL) divergence. The distribution ((g x)) is parameter- ized by the Gaussian parameters Hg and gg. In our implementation, LKL is used to avoid arbitrarily high-variance latent spaces of motion global token g.
The latent alignment loss Liatent directly minimizes the distance between the condition latent tokens Re (from the Condition Encoders) and the motion latent tokens (from the Motion Encoder). This encourages the information encoded from the two different modalities to align in the shared latent space. Including 2D condition encoder and 3D condition encoder alignment constraints:
Liatent = Lratent + Liatent 3D;
(10)
Ciaseut.c FMSE 2),
(11)
where FMSE denotes the mean squared error (MSE) loss.
The auxiliary loss Laux regularizes predicted motion de reconstructed from condition latent &e
Laux = Laux + Laux33-
(12)
Liatent_ Fr.
(13)
A.3 LATENT DIFFUSION MODEL
Architecture. The condition denoiser Go is implemented as a transformer-basee architecture con- sisting of 16 transformer layers illustrated in Figure 1. Each layer is configured with feed- forward dimension of 2048, a hidden dimension of 512, 16 attention heads, and the GELU activa- tion function. The latent space has dimensionality of 512, consistent with the Joint VAE. Follow- ing Yang et al. (2024), the diffusion timestep t is injected into the network through the modulation module of an adaptive LayerNorm. For temporal modeling, we apply Rotary Positional Encoding (RoPE) as temporal positional encoding to the hidden states. For vision encoding, we adopt the pretrained DINO-v2 Oquab et (2023) backbone, with weights kept frozen.
3D RoPE. We adopt Rotary Positional Encoding (RoPE) (Su et al., 2024), which has been shown to improve scalability and adaptability. RoPE encodes relative positional information through rota- tions in the complex space:
R, (x, m) = [co((m0:) sin(mo:) sin(m),) cos(m),)
22
(14)
122+1
where a is the input query or key representation, is the positional index, is the feature dimension index, and 0g is the frequency.
Given that the vision backbone extracts tokens with temporal length N, spatial height h. and width w, and to capture both spatial and temporal structures, we extend RoPE into a 3D formulation, following prior work (Kong al., 2024; Yang et al., 2024). The attention dimension is divided into three complementary subspaces, each dedicated to one axis. Independent sinusoidal embeddings are generated for the temporal, horizontal, and vertical dimensions, capturing relative positional information along each axis. Concretely. we compute the rotary frequency matrix separately for the coordinates of time, height, and width. The feature channels of the query and key are partitioned into three segments (dz, dr, dw,), and each segment is multiplied by the corresponding coordinate frequency. The outputs are then concatenated produce position-awar query and key embeddings, which are applied in attention computation. Compared to the standard RoPE, this 3D extension jointly encodes temporal continuity and spatial structure in a unified representation.
Losses.
The denoiser model is trained with the following losses:
Ldenoiser
Esimlee
Whec Lree-
(15)
We train the denoiser to predict the clean latent variable with the simple objective Simple- Training proceeds by sampling za from the dataset, applying the forward process obtain noisy latent at. predicting o0 using Go. and minimizing the reconstruction error. The simple objective is defined as:
Lximple (ttp:o/ctaooo.o........o........
())zo):
(16)
16

---

Published as a conference paper at ICLR 2026
Original Video Frame
Visible Hand
Rendered Hand Mesh
Complete Hand
Figure Illustration of hand occlusion level computation on Dex dataset.
where Go(=,, f, C)) denotes the predicted clean latent, and FMSE is distance function which is implemented using the mean squared error (MSE) loss.
The reconstruction loss Lie (same defined in Eq. (6)) encourages the predicted motion sequence & to remain close to the ground-truth sequence æ by jointly penalizing discrepancies in both MANO parameters and the regressed 3D joints.
Inference. At inference time, we initialize with Gaussian noise žr (O, I). The denoiser is applied iteratively, where at each step it predicts the clean latent &g and updates the noisy latent 2, towards a lower-noise state, until clean latent 20 is obtained. The final latent 2g then decoded by the autoregressive decoder in the Joint VAE to generate hand motion sequence dt
Benefiting from the design of the Joint VAE, structured control signals such as 2D and 3D keypoints are encoded into the shared latent space and can be directly fused with the noisy latent 24. Visual information extracted by the frozen vision backbone, processed through the hand-relevant atten- tion module, and represented as hand tokens, which are integrated into the denoiser at each step. We further adopt classifier free guidance (CFG), assigning an independent unconditional token to each control modality. This design enables flexible integration and combination of different condition inputs.
B EXPERIMENTAL SETUP
B. DATASETS
We train our model the Dex (Chao et al., 2021) and HOT3D (Banerjee et al., 2025) datasets, and additionally evaluate out-of-domanl generalization on HO3D (Hampali et al., 2020). To simplify learning, we horizontally flip input images and corresponding annotations whenever the targeted hand is left, resulting in right-hand-only network. Unless otherwise specified, UniHand trained exclusively the training splits of Dex YCB and HOT3D, and all reported results obtained from a single unified checkpoint, without dataset-specific fine-tuning or architectural modifications.
Since both Dex and HOT3D contain motion sequences, during training, randomly select valid initial pose within a sequence and sample consecutive frames to construct motions of length N 48. At the inference stage, the sequence length is required be integer multiple of the autoregressive decoding segment length. If this condition is not satisfied, we pad the sequence by repeating the control conditions the final frame.
DexYCB. Dex (Chao et al., 2021) is a large-scale dataset containing 000 videos of single- hand object manipulation. It features 10 subjects performing grasps on 20 objects from the YCB- Video dataset (Xiang al., 2018). Each action sequence captured by 8 synchronized RGB-D cameras from fixed third-person viewpoint. For evaluation, follow the official protocol and adopt the default split (S0) for training and testing.
To evaluate the degree of hand occlusion, we compute the ratio between the occluded hand region and the complete hand region. As illustrated in Figure 3, we obtain two types of masks: the visible hand mask Mvs. where only the non-occluded pixels of the hand are labeled as 1 (provided by the dataset), and the complete hand mask Mhant which is obtained by decoding MANO parameters and rendering the hand mesh, covering the entire hand region regardless of occlusion. Formally, the
17

---

Published as a conference paper at ICLR 2026
occlusion ratio is defined as:
Maadd
Mand O Mivs
Foce
(17)
Mhand
where LM| denotes the number of pixels labeled as in mask M. This metric allows us to categorize frames in the DexYCB dataset into different occlusion levels.
HOT3D, HOT3D (Banerjee et al., 2025) is a first-person dataset recorded with dynamic cameras, covering both single-hand and two-hand manipulations. It provides ground-truth camera trajectories as well as world-coordinate MANO annotations for each frame.
In experiments,
the HOT3D-Clips version, which consists of carefully selected sub- sequences from the original dataset. Each clip contains roughly 150 frames, corresponding to about 5 seconds of video. We adopt the subset collected with the Aria device and use only the main-view RGB images as vision conditions, since the Quest3 device does not provide RGB data. Ground- truth poses are available for every modeled object and hand in all frames. Because the official test split does not provide ground-truth annotations, we use the split based on the official training set, resulting 1, 272 clips for training and 244 clips for testing.
B.2 IMPLEMENTATION DETAILS
All experiments conducted 4 NVIDIA 80GB H800 GPUs. We adopt DeepSpeed (Rasley et al., 2020) for training to reduce memory consumption and improve efficiency. The AdamW (Loshchilov & Hutter, 2017) optimizer used with an initial learning rate of 1 10- scheduled with 100 warmup iterations followed by linear annealing.
We first train the Joint VAE. A small KL weight WKL. 1 x 10 is applied maintain an expressive latent space while preventing arbitrarily high-variance latent variables. The other loss terms are balanced with weights of wjoint ree 0.5 for the joint reconstruction loss, Whttet 0.1 for the latent loss, and Wanx 0.1 for the auxiliary loss. After training, the motion encoder, condition encoders, and autoregressive decoder are frozen. The latent denoiser is then trained using DDPM (Ho et al., 2020) with 50 diffusion steps and cosine noise scheduler. A weight of Whee 1.0 is applied during training.
At inference time, employ DDIM (Song et al., 2020) with 10 diffusion steps for efficient gener- ation while mitigating error accumulation, and set the CFG scale to w = 2. Following the ablation study, we adopt vision frames and 2D keypoints the default condition configuration, since 3D keypoints are not directly available in real-world scenarios. For 2D keypoint detection, we utilize the pre-trained ViT backbone from HaMeR (Pavlakos al., 2024), which is also employed for the initialization of the first-frame hand pose.
C VISUALIZATION
C.I HAND MOTION IN CAMERA COORDINATE SPACE
To further demonstrate the effectiveness of our method under challenging scenarios such as severe occlusions and temporally incomplete conditions, we present qualitative comparisons in Figure 5, Figure 6, and Figure We compare HaMeR (Pavlakos al., 2024) with our proposed UniHand. The visualizations show that UniHand reconstructs more temporally stable and geometrically plau- sible hand poses, particularly when the hand is heavily occluded or interacting with objects. These results indicate that our unified generative framework effectively leverages heterogeneous conditions to maintain robustness and fidelity in complex real-world scenarios.
C.2 HAND MOTION IN WORLD COORDINATE SPACE
In Figure 4, we include additional visualizations of generated hand poses and trajectories in world coordinates. The first example presents left-hand motion sequence, which illustrates how Uni- Hand maintains consistent predictions across both hands. As described the main text, UniHand horizontally flips input images and their corresponding annotations whenever the targeted hand is
18

---

Published as a conference paper at ICLR 2026
Frames
UniHand (Ours)
Groundtruth
Frames
UniHand (Ours)
Groundtruth
Figure 4: Additional visualization of generated hand poses and trajectories.
left, resulting in right-hand-only network that simplifies learning. Thus, the model always predicts right-hand MANO parameters, which also a standard practice adopted by prior methods such as HaMeR. For left-hand inputs, invert the flipping transformation the predicted right-hand MANO parameters to obtain the corresponding left-hand result.
D STATEMENT
D.1 THE USE OF LARGE LANGUAGE MODELS (LLMS)
We used Large Language Models (LLMs) only writing assistant for language polishing during the preparation of this paper. LLMs were not used in the ideation, experiments, data collection. or result analysis. The authors take full responsibility the content of this paper, including the text that was refined with the assistance of LLMs.
D.2 REPRODUCIBILITY STATEMENT
We have taken several steps to ensure the reproducibility of our work. In the supplementary material, provide the code for the proposed method, data loader, and inference pipeline. A detailed description of dataset preprocessing, splits, and statistics is included in Appendix B. I. Comprehen- sive model architectures and implementation details are presented in Appendix A and B.2. These materials, together with the released code, are intended to facilitate the reproduction of our results and further research on this topic.
19

---

Published as a conference paper at ICLR 2026
Frames
UniHand (Ours)
HaMeR
Figure 5: Qualitative comparison between HaMeR and our UniHand. Our method generates more continuous and accurate hand pose sequences compared to HaMeR.
Frames
UniHand (Ours)
HaMeR
Figure 6: Qualitative comparison between HaMeR and our UniHand. In cases of severe hand self- occlusion, HaMeR misclassifies the right hand as the left hand, resulting in poor reconstruction quality, whereas UniHand generates reliable and consistent hand motions.
Frames
UniHand (Ours)
HaMeR
Padladd
Fadiold
Pali
Figure 7: Qualitative comparison between HaMeR and our UniHand. HaMeR fails to estimate valid poses in video frames where the hand is absent, whereas UniHand maintains stable reconstructions by exploiting vision perception and temporal modeling.
20


Reviews:
[{"review_id": "HXiwurDvGX", "reviewer": "Reviewer_76ft", "summary": "The work seeks to unify the hitherto distinct domains of hand motion generation and hand motion reconstruction. This is achieved by a variational autoencoder embedding various non-visual modalities (hand trajectories, human body keypoints) into a shared latent space. In parallel, a hand perceptron extracts frame-wise features from visual inputs. The extracted non-visual latent representations are then used by a latent diffusion model to generate the hand pose sequence, with visually informative features being provided at every layer of the motion denoiser. The work compares the proposed method with multiple hand motion generation and reconstruction baselines on three datasets, consistently outperforming them.", "questions": "How does the method differentiate between the left and the right hand in its output?\nHow does the method perform in the presence of feet, which are often misdetected as hands in egocentric videos?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The proposed method shines when evaluated against numerous baselines, even in the presence of significant occlusion (Table 1). It is able to handle multimodal conditioning input, making it flexible and able to benefit from various types of known information at inference time.\nThe proposed method is thoroughly ablated with respect to its components and possible input modalities.\nThe work includes an honest discussion of its limitations.", "weaknesses": "The submission could benefit from more qualitative examples in the supplementary material. This is especially relevant for generative models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8UehyGZH4d", "reviewer": "Reviewer_LJ9z", "summary": "This paper aims to unify both hand pose estimation and generation using a single diffusion-based framework.\nThe proposed model, UniHand, handles both structured MANO and key points (as conditions for generation), as well as images (for estimation). \nSeparate encoders are used to bring different hand conditions into the same latent space, then the method runs denoising diffusion process in the latent space, aiming to to produce smooth 4D motion output.", "questions": "Apart from the mismatch between what is claimed and what is experimented, the writing of this paper is good in general.\n\nThe points that is unclear:\n\n1. Figure 1, bottom left, what are two red boxes represent at the bottom left? are they $g$?\n\n2. Shouldn't motion encoder  be called hand pose encoder? \n\n3. Line 207-208, better to say \"At each _autoregression_ step\" instead of simply \"At each step\", which can mean diffusion step.\n\n4. Line 883, DeepSpeeds need reference.\n\n5. Line 118, EASY-HOI is not relevant to Hand Motion Generation.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 1, "presentation": 3, "contribution": 2, "strengths": "This paper is well motivated: it aims to bring the estimation and the generation model together. \nThe proposed model indeed can do these two task in a unified way. \nThis paper does not propose a new problem formulation, but the effort to introduce an unified solution for two problems are interesting and valuable.\n\nThe writing is clear in general: we can understand how the proposed framework achieve the proposed goal at the high level.", "weaknesses": "The main issue of this paper is it does not address the proposed goal: unifying both estimation *and generation*. The generation ability of the framework is not tested/reported. \nThe paper is motivated by unifying the estimation and generation into the same framework. The motivation is sound, and the de-noiser in the framework is indeed a generative model. However, all results are reported **only on the hand pose estimation task**. \nThe sole focus of estimation deviates from what is described in the title, abstract and intro, as well as in the related works where the authors talk about unifying generation under their framework in Line 134-136.\nThe authors could have shown their results on any generation problems, e.g. [Zuo et al., 2023] in Line 128 or [Zuo et al., 2024] in Line 115-116 of the grasp generation problem. I understand that extending the system for these generation tasks can involves much more work, but since the authors claims the unified method, one would expect the generation task results.\n\nIn Line 347-348, the authors proposes to evaluate the generation via estimation:\n> Hand pose estimation in the camera coordinate space provides the most direct way to evaluate the quality of motion generation conditioned on visual observations.\n\nbut above is saying that the estimation and the generation are the same task, which contracts what the authors say in Line 013 and throughout the whole introduction!\n\nThe author should really test their framework on a few tasks listed in their Sec 2.2, without the results on the generation task, otherwise this paper gives a wrong impression to the audience. \n\nRegarding the estimation evaluation setup, it is unclear to me when $c_{2D}$ and $c_{3D}$ are provided to the model? In my understanding, the standard estimation task setup does not provide $c_{2D}$ and $c_{3D}$ to the model.\n\nSince providing experiment results on the real generation task will lead to substantial changes to the paper and is beyond what can be done during rebuttal, I suggest rejection.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3E6d4GWGDC", "reviewer": "Reviewer_4cS2", "summary": "This paper presents UniHand, a unified diffusion-based framework for 4D hand motion modeling that jointly addresses motion reconstruction and generation. It reformulates both tasks as a conditional motion synthesis problem, bridging the gap between estimation and generation. A Joint Variational Autoencoder (Joint VAE) aligns heterogeneous conditional inputs such as 2D and 3D keypoints and MANO parameters into a shared latent space, enabling robust modeling under incomplete or inconsistent data. The Hand Perceptron module extracts hand-specific features directly from full visual inputs without requiring explicit detection or cropping, while a canonical coordinate system ensures spatial and temporal consistency under dynamic camera settings. Experiments on DexYCB, HO3D, and HOT3D datasets demonstrate that UniHand achieves state-of-the-art results, especially under severe occlusion or partial input, validating its effectiveness and generalization for unified hand motion estimation and generation. However, despite its strong performance, the method suffers from algorithmic complexity and high computational cost, and the exploration of the unique challenges in multi-modal alignment for 4D hand pose estimation remains insufficient.", "questions": "- While UniHand demonstrates strong performance, the diffusion-based generation pipeline and multimodal latent alignment introduce high computational cost. Can the authors quantify the training and inference time compared to existing methods, and discuss possible simplifications or acceleration strategies?\n- The paper acknowledges that UniHand struggles to maintain globally consistent trajectories under large camera movements due to the lack of explicit camera extrinsics. Could the authors elaborate on potential ways to address this—e.g., by incorporating implicit camera modeling or scene-aware constraints?\n- Although UniHand integrates visual, 2D, and 3D conditions through a joint VAE, the paper could further investigate how multimodal alignment contributes to 4D hand pose estimation quality. Would ablation or visualization of the latent space help clarify how modalities interact and which provide the most benefit?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper proposes a unified diffusion-based framework that integrates both estimation and generation for 4D hand motion modeling, offering a fresh formulation of conditional motion synthesis that extends beyond task-specific designs.\n- The technical design, including the Joint VAE and Hand Perceptron modules, is well-motivated and validated through comprehensive experiments on multiple datasets, showing robustness under occlusion and dynamic camera motion.\n- The paper is clearly written and systematically structured.", "weaknesses": "- Real-world deployment may be limited without efficient preprocessing of input modalities.\n- Heavy computational and data requirements for training.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)