<figure><img src="image_1.png" /></figure>

HAL
open science

# Plausible and Diverse Human Hand Grasping Motion Generation

Xiaoyuan Wang, Yang Li, Changbo Wang, Marc Christie

► To cite this version:

Xiaoyuan Wang, Yang Li, Changbo Wang, Marc Christie. Plausible and Diverse Human Hand Grasping Motion Generation. 2024 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), Mar 2024, Orlando, France. IEEE, pp.1007-1008, 2024, 10.1109/VRW62533.2024.00300. [hal-05239372](https://hal-INRAA.hal science/hal-05239372v1)

HAL Id: hal-05239372

https://inria.hal.science/hal-05239372v1

Submitted on 4 Sep 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Plausible and Diverse Human Hand Grasping Motion Generation

Xiaoyuan Wang*

Yang Li†

Changbo Wang‡

Marc Christie§

University of Rennes, Inria,
CNRS, IRISA

East China Normal University

East China Normal University

University of Rennes, Inria,
CNRS, IRISA

East China Normal
University

<figure><img src="image_3.png" /><figcaption>Figure 1: Our system generates diverse grasping motions (different starting positions and reaching angles) for the targeted object.</figcaption></figure>

## ABSTRACT

Techniques to grasp targeted objects in realistic and diverse ways find many applications in computer graphics, robotics and VR. This study generates diverse grasping motions while keeping plausible final grasps for human hands. We first build on a Transformer-based VAE to encode diverse reaching motions into a latent representation noted as GMF and then train an MLP-based cVAE to learn the grasping affordance of targeted objects. Finally, through learning a denoising process, we condition GMF with affordance to generate grasping motions for the targeted object. We identify improvements in our results, and will further address them in future work.

**Index Terms:** Human-centered computing—Human-computer interaction (HCI)—Motion Generation

## 1 INTRODUCTION

Hands are obviously crucial for humans when interacting with objects in the world. In the fields of virtual reality and robotics, the study of hand grasping has been a long-standing research endeavor. The analysis of real hand motions is essential for collecting information about the complex nature of the interactions of hands and objects, and to assist the design of general reaching and grasping actions on robotic systems.

In the context of grasping, one can divide the process into two distinct phases: the reaching phase and the closing phase. The reaching phase involves moving the hand close to the object to be grasped, while the closing phase focuses on the actual hand motions that cause not only contact between hand and object but also ensure a certain grasp quality synonym of physical plausibility.

Regarding the reaching phase, different initial hand positions lead to different reaching motions. For one object, the final grasping pose can be located on various parts of its surface, which we know as the *grasp affordance*. We try to design a method that enables learning the grasping motion generation while embedding grasp affordance to be the condition of the learning properly.

In recent years, generative models such as VAE and Diffusion have witnessed rapid development and have achieved significant breakthroughs. The key point of the VAE model is that it not only encodes input data into a latent representation but also imposes a probabilistic structure on this encoding. On the other side, Diffusion models estimate the conditional probability of the data given the noisy observations where noise is added to the data progressively. As a result, Diffusion models are able to account for the distribution of the complex input data.

In this study, we learn to generate grasping motions by combining VAE and Diffusion models. First, We learn a Transformer-based VAE to encode diverse reaching motions into a basic latent representation as Grasping Motion Field (GMF), Then, we embed the targeted objects to be grasping affordance which is latent by training an MLP-based cVAE. Finally, through learning a denoising process, we condition GMF with affordance to enable sampling of GMF with the condition of the targeted object to generate grasping motions. Our results show improvements in both the diversity and grasp quality of generated motions.

## 2 METHOD

We explain our method of three modules: the Grasping Motion Filed (GMF), Grasp Affordance, and Diffusion model.

### 2.1 Grasping Motion Field (GMF) and Grasp Affordance

We introduce a Transformer-based VAE architecture to learn a GMF, which is composed of an encoder and a decoder. We train this VAE on the grasping motion sequences $x \in \mathbb{R}^{L \times 51}$ that are composed of $(R_M, \theta_M, t_M)$ and variable length $L$.

Like most VAE-type models, our loss includes two parts: the Reconstruction loss and the Kullback-Leibler (KL) loss. The reconstruction loss indicates the reconstruction ability to ensure the

<footnote>*e-mail: xiaoyuan.wang@irisa.fr</footnote>

<footnote>†e-mail: yli@cs.ecnu.edu.cn</footnote>

<footnote>‡e-mail: cbwang@cs.ecnu.edu.cn</footnote>

<footnote>§e-mail: marc.christie@irisa.fr</footnote>
model better represents the training data by encoding it into a lower-dimensional latent space representation and then decoding it back into the original input space. In our method, we design $\mathcal{L}_{rec}$ on MANO parameters and joints positions:

$$ \mathcal{L}_{rec} = w_R * \mathcal{L}_{R_M} + w_{\theta} * \mathcal{L}_{\theta_M} + w_t * \mathcal{L}_{t_M} + w_j * \mathcal{L}_{J} \quad (1) $$

where $w$ is the weight of every loss item, $J$ is the hand joints sequence composed of 21 joints coordinates with $T$ frames. We achieve every reconstruction loss item using the Mean Squared Error (MSE) function. The KL loss is the regularization term to encourage the VAE to learn a specific type of latent space structure, which maps data space to the latent space and enables the sampling from the learned distribution not discrete points.

Our Grasp Affordance module is a cVAE model including Encoder and Decoder. In the training stage, the input is ground truth motions and the condition is the targeted object. We use reconstruction loss and KL loss similar to the above. In the inference stage, we take the targeted object and the sample of Gaussian Distribution as input of the Decoder and further embed the generated grasp affordance to be latent through the Encoder.

## 2.2 Diffusion Model

Our conditional motion latent diffusion model is composed of a denoising model $\epsilon_{\theta}$, forward process and reverse process working on GMF $z_0$ similar to [2]. In the forward process, $\epsilon_{\theta}$ learns $\theta$ and converts GMF to Gaussian. We take $z_0$ and apply standard Gaussian noise $\epsilon$ to it progressively according to a noise schedule and an update rule to get noisy $z_t$, in which $z_t$ is sampled from arbitrary step $t$. The training objective of the model $\epsilon_{\theta}$ is to predict the added noise of $z_t$ with the condition $c$ sampled from grasp affordance latent $A \in \mathbb{R}^{1 \times d}$. The loss is calculated by $\mathcal{L}_{e} = \text{MSE}(\epsilon_{pred}, \epsilon)$, where $\epsilon_{pred} = \epsilon_{\theta}(z_t, t, c)$.

While the forward process converts GMF to Gaussian, the reverse process takes input from Gaussian samples and outputs GMF samples. At inference time, the denoising process takes some steps from $z_t$ to generate $z_{t-1}$ using the formula in [4].

# 3 EXPERIMENTS

## 3.1 Datasets and Evaluation Metrics

DexYCB [1] is a human hand grasping dataset including reaching motions. We evaluate our method by running inference 20 times for every unseen testing object instance with the metrics of Diversity and Grasp Quality. For the Diversity metrics, we adopt Frechet Inception Distance (FID) to measure the similarity between the distribution of real data and generated data in feature space. Note that we train our own feature space by designing a Gated Recurrent Units (GRU) model. FID last computes the FID only for the last frames. Multimodality (MM) measures the dissimilarity between pairs of generated samples per condition. For the Grasp Quality metrics, we adopt grasp Score which is a "force-closure" metric [3] that indicates, when its value is positive, that the grasp can resist some extent external force or torque applied to the object. In other words, it's an approximation of the minimum force to be applied to break the grasp stability. It therefore measures the physical stability of the grasp. When contact occurs, finger collision might also exist, since more finger surface results in more stable and effective forces, which needs to be balanced with the Score metric. So we specify the average number of Intersection hand vertices (Inter_r) to consider the contact surface between fingers and objects. We also report the Valid Ratio (Valid_r) which computes the overall ratio of those Score values bigger than 0 to further evaluate the plausibility of the generated motions and the sampling efficiency of our method.

## 3.2 Baseline and Ablation

We choose [5] as baseline marked as CGF, which learns an MLP-based cVAE to generate grasping motions. The differences between

<table><caption>Table 1: Diversity quality evaluation. The bold values mean the best result and the values with underline denote the second-best result.</caption><thead><tr><th>Method</th><th>FID↓</th><th>MM ↑</th><th>FID last ↓</th></tr></thead><tbody><tr><td>CGF [5]</td><td>21.210</td><td>10.748</td><td>0.606</td></tr><tr><td>ablation</td><td><b>16.261</b></td><td><u>10.937</u></td><td><u>0.532</u></td></tr><tr><td>Ours</td><td><u>16.622</u></td><td><b>11.857</b></td><td><b>0.294</b></td></tr></tbody></table>

<table><caption>Table 2: Grasp quality evaluation.</caption><thead><tr><th>Method</th><th>Inter Ratio ↓</th><th>Score ↑</th><th>Valid Ratio ↑</th></tr></thead><tbody><tr><td>CGF [5]</td><td>0.13</td><td>0.06</td><td>0.19</td></tr><tr><td>ablation</td><td>0.45</td><td>0.08</td><td>0.24</td></tr><tr><td>Ours</td><td><b>0.11</b></td><td><b>0.09</b></td><td><b>0.29</b></td></tr></tbody></table>

the baseline and ours hold in 3 aspects: (1) we adopt a Transformer-based architecture in VAE, (2) our object embedding is latent grasp affordance while only geometry features for baseline, and (3) we separate motion learning and conditioning by incorporating a VAE and a Diffusion model.

In our experiments, we also design a basic architecture, a cVAE based on Transformer architecture noted as ablation. The difference between this ablation and our method lies in how to make the model aware of the object to be grasped. For ablation, the condition is object geometry features and concatenated with input data. For our method, the condition is latent grasp affordance and fused in noise learning and denoising.

## 3.3 Analysis

### Comparison with diversity quality metrics

In Table 1, ablation shows the best result of FID and our method achieves the best results of other metrics, which indicates our method can generate more diverse motions while following the accurate motions features. Besides, ablation is better than the baseline method on motion diversity but not the last grasping pose.

### Comparison with grasp quality metrics

For our hand grasping motion generation task, the grasp quality of the last frame is important to measure the plausibility of the generated motions. In Table 2, Our method achieves the best results of all the metrics, especially for Valid Ratio. The results prove that our method is more stable in generating plausible grasping motions.

# 4 CONCLUSION

To learn how to generate diverse and plausible hand grasping motions, we proposed to couple VAE and Diffusion models in this work. And results show our method improves the quality and diversity of generated motions.

# REFERENCES

[1] Y.-W. Chao, W. Yang, Y. Xiang, P. Molchanov, A. Handa, J. Tremblay, Y. S. Narang, K. Van Wyk, U. Iqbal, S. Birchfield, J. Kautz, and D. Fox. DexYCB: A benchmark for capturing hand grasping of objects. In *IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, 2021.

[2] X. Chen, B. Jiang, W. Liu, Z. Huang, B. Fu, T. Chen, and G. Yu. Executing your commands via motion diffusion in latent space. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 18000–18010, 2023.

[3] C. Ferrari, J. F. Canny, et al. Planning optimal grasps. In *ICRA*, vol. 3, p. 6, 1992.

[4] J. Song, C. Meng, and S. Ermon. Denoising diffusion implicit models. *arXiv preprint arXiv:2010.02502*, 2020.

[5] J. Ye, J. Wang, B. Huang, Y. Qin, and X. Wang. Learning continuous grasping function with a dexterous hand from human demonstrations. *IEEE Robotics and Automation Letters*, 8(5):2882–2889, 2023.