Joint Diffusion for Universal Hand-Object Grasp Generation
Authors: Jinkun Cao
Jingyuan Liu
Kris Kitani
Yi Zhou
Venue: Accepted by TMLR ()

Abstract: Predicting and generating human hand grasp over objects is critical for animation and robotic tasks. In this work, we focus on generating both the hand and objects in a grasp by a single diffusion model. Our proposed Joint Hand-Object Diffusion (JHOD) models the hand and object in a unified latent representation. It uses the hand-object grasping data to learn to accommodate hand and object to form plausible grasps. Also, to enforce the generalizability over diverse object shapes, it leverages large-scale object datasets to learn an inclusive object latent embedding.  With or without a given object as an optional condition, the diffusion model can generate grasps unconditionally or conditional to the object. Compared to the usual practice of learning object-conditioned grasp generation from only hand-object grasp data, our method benefits from more diverse object data used for training to handle grasp generation more universally. According to both qualitative and quantitative experiments, both conditional and unconditional generation of hand grasp achieves good visual plausibility and diversity. With the extra inclusiveness of object representation learned from large-scale object datasets, the proposed method generalizes well to unseen object shapes.

Published in Transactions on Machine Learning Research (11/2025)
Joint Diffusion for Universal Hand-Object Grasp Generation
Jinkun Cao* Carnegie Mellon University
jindunceanddrew.. www.ewwwwwwm..oommmmmmmmmmm edbu
Jingyuan Liu Adobe
jingyliu@udobe.ccm
Kris Kitani Carnegie Mellon University
kmbkilaniS@nddre.coom cra. eduu
Yi Zhou Roblor
yishou@roblors com
Reviewed on OpenReview:
https: /
// openreview. net/ forum? id= T20stsYR6a
Abstract
Predicting and generating human hand grasp over objects is critical for animation and robotic tasks. In this work, we focus on generating both the hand and objects in grasp by single diffusion model. Our proposed Joint Hand-Object Diffusion (JHOD) models the hand and object in a unified latent representation. It uses the hand-object grasping data to learn to accommodate hand and object to form plausible grasps. Also, enforce the generalizability, over diverse object shapes, it leverages large-scale object datasets to learn an inclusive object latent embedding. With without given object conditional to the object. Compared to the
optional condition, the diffusion
model can generate grasps unconditionally usual practice of learning object-conditioned grasp generation from only hand-object grasp data, our method benefits from more diverse object data used for training to handle grasp generation more universally. According to both qualitative and quantitative experiments, both conditional and unconditional generation of hand grasp achieves good visual plausibility and diversity. With the extra inclusiveness of object representation learned from large-scale object datasets, the proposed method generalizes well to unseen object shapes.
1 Introduction
In this paper, explore the generation of hand-object grasps. Unlike traditional approaches that fit to limited set of objects found in 3D hand-object interaction datasets, goal is to generalize across diverse object geometries. We develop a joint diffusion model capable of generating hand poses either conditionally on or jointly with object shapes. Our method approaches objects from a purely geometric perspective, eliminating the need for language descriptions or category labels. Despite having access to limited hand-object interaction datasets, our model learns a comprehensive object shape embedding by leveraging large-scale object shape datasets. It generates paired hand and object grasps by denoising a joint latent representation.
To generate hand grasp over objects, existing works Taheri et al. 2020 Lu et al. 2023 Fan et al.
2023
typically rely on datasets with full-stack 3D annotations (Taheri et al.) 2020 Fan et al. 2023 Yang et al. 2022) from object scans and hand pose capture. However, with the difficulty of capturing 3D object models and hand gestures, this area of research faces notorions limitations annotated data,e.g,, only dozens of
Jinkun and Yi were affilated with Adobe Research when the main part of the work conducted

---

Published in Transactions on Machine Learning Research (11/2025)
Dblect-Conditione Grasp Generation
Object and Grasp Joint Generation
Grasp-Conditioned Image Generation
Figure/ 1: Applications of our proposed method Joint Hand-Object Diffusion (JHOD). Left: generating the hand grasp unseen object. Middle: jointly sampling combination of the hand and the object for plausible grasp. Right: the generated grasp be used guidance for generating photo-realistic grasping images using existing image generation tools (Adobe 2024
object shapes are available in dataset (Fan et al. 2023 Taheri et al. 2020 Morcover, existing datasets are designed for different hand parametric models (Miller & Allen 2004 Jian et al. 2023 shal Wang al.,
2023 Romero et al. 2022 Taheri al. 2020 Yang al.] 2022), which prevents directly combining different data resources scale up. The limited scale of object annotations causes the method to overfit to biased object shapes and thus bad generalizability of generating grasps on unseen objects.
Humans have the remarkable ability to transfer grasping patterns to different objects and make plausible grasping choices for unseen objects based solely on their geometry. While prior knowledge, such as object category or functionality, can aid in manipulation, it is often not essential for determining grasp.
We refer to this capability as universal grusp generation. We argue that the key to universally generating hand-object grasps lies in developing inclusive object shape embedding. This is unattainable if train object embeddings using datasets with limited quantity and diversily. Although comprehensive 3D amno- tations are scarce, we have access to large-scale 3D object datasets. In this work, we investigate whether those datasots can be leveraged to enhance hand-object grasp generation by jointly modcling hand and object representations in a latent space. Our model is designed to accommodate partial supervision during training, accepting available data such as hand poses or object shapes. This flexibility significantly broadens the range of datasets that can be utilized for learning. Our model can generate complete 3D grasping scenes, including both the posed hand and object, a posed hand conditioned on given object geometry. We believe that this flexible approach to generating hand-object interactions, without relying on domain-specific auxiliary information, represents a significant step toward human-like universal grasp generation.
Following such an intention, we propose our Joint Hand-Object Diffusion (JHOD) model. It follows the latent diffusion model (Rombach al. 2022) (LDM) encode and ensemble different modalities into a latent space and then generate plausible samples by the probabilistic denoising process (Ho et al. 2020). To learn from more diverse objects, we construct the model by leveraging the object shape encoding and decoding networks pre-trained the large-scale generic 3D object shape dataset (Chang et al. 2015). As we aim to derive a universal grasp generation solution, we remove the category-specific information when constructing the latent code for objects and encode the object shapes in a purely geometric fashion.
As the hand articulation and positions are always coupled to the object shape in hand-object grasping, we design the latent diffusion to denoise the latent code from different modalities as whole. We decouple each modality representation allow model training by combining data from different resources with het- erogeneous annotations. Therefore, method has independent encoder and decoder networks for different
2

---

Published in Transactions on Machine Learning Research (11/2025)
modalities and can optimize the generation of each modality solely. Such design relieves the limitation of object diversity in hand-object interaction datasets as we could leverage the data from generic object shape datasets to tune the object generation part. The model is trained to generate the hand part regarding the corresponding object shape embedding, so an inclusive object generation improves the corresponding hand grasp generation on diverse object shapes. To achieve disentangled modality representations to boost training. we propose to use asynchronous denoising schedulers for different modalities during noise diffusion and denoising in training.
In conclusion, we have developed joint hand-object grasp generation model capable of sampling from either the joint distribution of both hand and object or the object-conditooned distribution for hand generation. This model effectively leverages training resources with varying annotations. Through both qualitative and quantitative evaluations, our method demonstrates strong performance in grasp generation for both unconditional and object-conditooned scenarios. Thanks to the posterior generation learned from extensive data resources, our model significantly outperforms others in generating grasps for out-of-domain object shapes, crucial aspect of universal grasp generation. Given that no existing works have supported the end-to-end generation of both hand and object to form grasp, we believe our approach is pioneering in the field of universal grasp generation.
With the flexibility to generate both modalities in hand grasp, our model can facilitate many downstream applications. The grasps are good references for generating photo-realistic images with good geometry alignment and siguificantly fewer artifacts. We first generate grasps in 3D and use them as conditional signals for image generation (Adobe) 2024) can help produce high-quality hand images with challenging gestures and view angles. We demonstrate using grasp from our model the condition improves grasp aligmment on images and relieves artifacts. Some examples of object-conditioned grasp generation, joint generation, and image generation with grasp rendering are presented in Figure [ The main contribution of this work is to propose a generative model capable of generating hand grasp in either unconditional or object- conditioned fashions and the corresponding stratogy to enhance the performance with limited available 3D hand-object grasp annolations.
2 Related Works
Hand Grasp Generation. Modern generative models (Ho et al. 2020 are recently introduced into hand grasp reconstruction (Ye et al. 2023a) and generation. Hand grasp reconstruction asks for hand grasp ren- dering aligned with images while hand grasp generation asks for hand grasp visually physically plausible, GrabNet (Taheri et al.) 22020) is a commonly used method for hand grasp generation but it can hardly be generalized to diverse object shapes. HOIDiffusion /Zhang et al.) 2024b) uses the 3D) hand grasp from GrabNet and an image diffusion model to generate hand-object interaction images. AffordaancDDfftusion YYo et al. 2023b) also studies the generation of hand-object interaction for image synthesis. Instead of generating hand-objes interaction images or videos, we focus on improving hand-object interaction in 3D space. Many
recent works are based on diffusion models (Ye et al.] 2024 Lu et al.
2023
Christen et al. 2024) physics-
based simulators (Zhang et al. 2024a Luo et al. 2024a
Xu et al.
2023
Luo et al.) 2024b). Compared to
existing works which typically require given object Taahri et al.
2020:
Zhang et al. 2024b Ye et al.
2023b; Zhou et a (2024) shape as input, we desire generating a hand grasp over a given object or generating both the object and hand to form a plansible grasp.
Multi-Modal Generation. With the rise of diffusion models, multi-modal generation has been extended in many areas, such as image+text generation (Bao et al., 2023) and audio+pixel generation (Ruan et al, 2023). Our model can jointly generate the hand and object in grasp, making another type of multi-modal generation. A concurrent work UGG (Lu et al. 2023 studies to generate both hand and object to form grasp but it uses the ShadowHand parametric model to leverage the large-scale synthetic ShadowHand grasping datasets (Wang et al.] 2023) and the final results are optimization-basel instead of directly from the generative model. On the other hand, another concurrent work G-HOP (Ye et al., 2024) builds a multi- modal hand-object grasp prior by encoding object shape and hand poscs into a unified latent space but a language prompt is required to provide necessary conditional information. In this work, we focus on a dual-modal latent diffusion model to generate the modalities of 3D object and hand at the same time
3

---

Published in Transactions on Machine Learning Research (11/2025)
Prior
unconditional generation
Object Decoder
Sample
+
ZT
Denoise
Zo
+
Hand Decoder
Obj Enc.
object condition (optional)
conditional generation
The illustration of proposed Joint Hand-Object Diffusion (JHOD). We present the main modules of the model while removing the secondary modules for simplicity, The optional object condition can turn the generation into object-condiioned.
Figure 2:
without requiring optimization auxiliary conditions. The hand grasp is purely geometric-based. It allows more flexibility for downstream tasks, such as generating photo-realistic images by using the rendering of generated 3D hand-object interaction as a condition.
3 Method
In Section 3.1 we first provide the formal problem formulation. We then review the related preliminary knowledge for methodology and data representation in Section 3.2 Finally, we introduced our proposed method in Section 33
3.1 Problem Formulation
An instance of hand-object grasp consists of a posed hand, denoted as H, and a posed object, denoted as O. To generate hand-object grasp unconditionally, we model the joint distribution of hand and object:
(0, HJ Da.
(1)
Also, the community of visual perception, animation, and robotics, people are interested in generating hand grasp conditioned on certain objects, which can be formulated as
(H) Dolo,:
(2)
where Og a provided object shape. In the previous works, the unconditional generation has been under- explored. The methods for conditional hand grasp generation usually required with no flexibility for object shape generation. They suffer from bad robustness to different object shapes. This is because the existing datasets with 3D annotations of both modalities due to the expansiveness of scanning the object shape and capturing the corresponding hand poses. These methods' limitation is underestimated because the object shapes used in training and testing in existing datasets share similar scales and geometries. In this work, we aim to solve both unconditional and object- conditioned grasp generation by single model. We also wish enhance the robustness of object shapes by deriving a more inclusive object embedding which also improves the generalizability of the hand part generation.
notoriously limited in covered object shapes

---

Published in Transactions on Machine Learning Research (11/2025)
3.2 Preliminaries
3.2.1 Latent Diffusion
Our method follows the latent diffusion models (LDM) (Rombach et al. 2022 Sohl-Dickstein et al. 2013 to generate samples in two modality spaces, i.e., articulable hands, and rigid objects. LDM diffuses and denoises in a latent space instead of on the raw data representations.
To diffuse data sample, given clean latent code 2g from data sample y Dys ie., 20 E(y) D2, we add noise following Markov noise process:
E [1,7].
where a E (0, 1) are constant the typical diffusion model (Ho et al., 2020) trains denoiser to predict the noise added per time step during
determined by the time step and noise scheduler. Though
the diffusion stage, there is another line of diffusion models (Ramesh et al. 2022 Tevet et al. 2022 that learns to recover the clean data sample directly. The loss turns to
LLDM == Exp-D (-1.. [1120 - Go (z, t112
(4)
requiring a generative model G$ to predict a clean latent code conditioned to a noisy latent code on a corresponding time step. We follow this paradigm of training diffusion models in this work. Besides the diffusion model itself, we also need the conversion between the raw data representation and the latent code. After recovering clean code fo, we can convert to the raw data representation by a decoder network X' which is trained to be the inverse of &, i.e. ideally (=0) Dy- Therefore, there are three modules with learnable weights: an encoder network &, decoder network X, and the denoiser network Ga.
3.2.2 Modality Representations
Here we introduce the raw representations of the modalitics involved in interrupting a hand-object interaction.
Hand. We follow the MANO (Romero et al., 2022 parametric model to represent hands. Compared to another parametric model ShadowHand (sha),, MANO is designed for animating non-rigid real human hands instead of robotic hands thus better fit for animation and photo-realistie generalization tasks. However, its higher complexity causes difficulty in grasping data synthesis. Therefore, there is hand grasping dataset as large-scale as the synthetic ShadowHand-baseet datasets (Wang et al. 2023 yet. This prevents research on the joint generation of MANO hands and objects in a straightforward supervised learning fashion. In this work, we directly use MANO parameters as joint rotation angles, i.e. f E R248 and the translation hands tú E RR, We use constant mean shape parameters 3o = o during linear blend skinning. The parameters of hands are concatenated and encoded by a hand encoder En to be the hand latent code Yи E R151
MlANO-basee
Object. For object representation, use the point cloud latent code yo = h E Nx4 where N is the number of points. As the grasp is modeled in the object-centric frame, no translation or oricntation is required in the object pose and shape representation. This convention is aligned with the definition of data in many hand-object grasp datasets, such as OakInk (Yang et al. 2022
3.2.3 Object Generation
The limited amount and high category-specific bias of the objects in hand-object interaction (HOI) datasets prevent learning generating grasps robust and generalizable to diverse object shapes. When training these datascts only, the learned object embedding is usually biased and overfit. By leveraging the largo-scale object-only 3D datasets, we can derive a more robust. object encoding for the grasp generation than learning from the HOI datasets solely. For the object encoding part, we borrow the point-cloud- based object shape gencration method LION (Vahdat al., 2022). Existing HOI generation methods can only learn object shapes from HOI datasots, including only dozens of objects, while LION learns from a much larger basis, i.c., more than 50,000 objects in ShapeNet (Chang et al. 2015). LION generates objects in two stages. In the
5

---

Published in Transactions on Machine Learning Research (11/2025)
first step, it derives a global latent code 2 E RD from a posterior distribution go (0(2g (P) where P R is the object point cloud. Then it samples a point cloud latent ho from posterior gEo (ho 2 LION generates the object shape by reverse process from sampled latent code
Peonin (P, ho: z - pxo Plo, z 2()ps (o))pohhoo 1 )R, (2
where Ko is the decoder network, v is the generator of point cloud latent code and y is the distribution of global latent code. In this work, remove the global latent code from LION to consider the object shape from a purely geometric perspective. We encode and decode the object between point clouds and latent code from the point cloud latent ho only. We convert the object generation part to
PEo. (P.. (o. = Pxo
(6)
where &' is fine-tuned from & by replacing the global prior y with a standard Gaussian. Thanks to the pre-training from large-scale object datasets (Chang al.) 2015), could use the weights of & and Xo from LION directly. By freezing the encoder and decoder, we fine-tune the object prior distribution with the object shapes from HOT datasets.
3.3 Joint Hand-Objec Diffusion (JHOD)
We now introduce our proposed Joint Hand-Object Diffusion (JHOD). There are two main goals. First, we would like to use the training data with heterogeneous annotations so that we can go beyond the limited object shapes in HOI datasets. Sccond, we want a single model capable of object-conditonned hand grasp generalion and unconditional hand-object joint generation. We present the overall illustration and key designs of our proposed method in Figure
3.3.1 Latent Codes for Different Modalities
For each modality, we have an MLP-based encoding network to convert the raw representation into latent codes:
2 = En(yH),
zf = Eolyo),
(7)
with the dimensions z R amd 2 E R 4x512 the whole hand-object interaction configuration
They are concatenated to be the final latent code for
20 2 e 0 E 5x512
(8)
This design allows us to jointly process different modalities without unnecessary entanglement. So we can train the model with partial modality supervision without corrupting the parameters for other modalities.
3.3.2 Asynchronous Denoising Schedulers
We propose generate either both hand and object or only the hand given an object shape. To realize this, we need a certain degree of entanglement between the two modalities as we need them cooperate to form valid grasp. However, we still desire certain disentanglement so that we could supervise the training with partially annotated data, e.g., 3D object shape data. In the usual fashion of organizing multi-modal latent codes in latent diffusion models, the latent codes of different modalities always have the same scale of noise corruption as they share the noise scheduler. In contrast, we desire that the two modalities can be denoised with respect to an arbitrary level of noise in the other modality. Therefore, we propose to use asynchronous denoising schedulers for this purpose during training. Instead of using a single noise scheduler, we have two schodulers tu and to to control the noise patterns in the hand latent code and the object latent code (3) we derive the corrupted latent codes
respectively. Following Equation
(9) with the same time step range lo, LH E []. This design allows the diffusion to deal with the noise in each modality separately. During training, it allows using object-only data to supervise the object part only. During sampling, it allows the model generate only the hand given an object shape as the condition. Similar to baod023miiifffser, we will learn the joint distribution for hand-object pairs and the marginal distribution for the hand at the same time in an end-to-end fashion by manipulating the time schedulers.
H ziz = 11"z,((z) H1 [ 2rn),, zio = 1122 12.
6

---

Published in Transactions on Machine Learning Research (11/2025)
3.3.3 Training and Sampling
Training. When training on HOI data samples, with the asyuchronous noise schedulers, we can derive the assembly of two corrupted latent codes as
4o i + zio.
(10)
Due to the clement-wisc property of MSE loss, we can calculate and back-propagate the gradient to certain part of the latent codes independently. We leverage this property to train the unconditional and conditional
generation at the same time. To learn the joint distribution, we apply the unconditional generation loss
Luncoud. = E Doro-Dorr (t), ||20 Go(Etwnto Lu, (o,(3))
(11)
where Dhoi is the distribution of the HOI datasots TTheri et al. 2020 Yang et al., 2022). We also train
the model to learn the object-conditioned grasp generation. Given an object latent code zn we set l0 = 0 indicating that the object part is noise-free free and serves as a condition to derive the marginal distribution. The object-conditooned generation loss thus
Leond. = -bp--Door, [1 - Go mm)
(12)
with ztn = =ix ( z2 The conditional and unconditional generation losses require uncorrupted and corrupted object latent codes respectively. In practice, we combine these two training objectives in a 1:1 ratio for a single draw of training data batch. This training strategy is similar to the classifier-Iree guidance (Ho & Salimans| 2022) for diffusion model training. However, the optional condition (object) exists in the input and output instead of in an independent condition vector.
=
Finally, we could leverage the large-scale 3D object shape datasets train the object distribution only. This learning goal is designed to improve the object latent code by exposing it to broader distribution of object shapes. We seleet object samples from both HOI datascts DHOi and object-only datascts Do. The loss turns to
Lobj. = Fg
- -DD-Dooor.too ||:00 - Go 1o)1
(13)
where we apply no supervision on the hand branch. While a hand latent code is present as a part of the joint latent code, is randomly initialized and remains untrained. The latent codes only take the object part as the variable:
2o = 20 Zio = l., s.l. N(0,1)
(14)
Obviously, the gradient only influences the object-related parts while keeping the hand part as-is. During training, the object encoder and decoder networks are pre-trained on the ShapeNet (Chang et al.] 2015 and frozen. All other encoder, decoder, and denoiser networks are trained end to end.
Similar to the proximity sensor features (Taheri et al. 2024) and the Grasping Filed (Karunratanakul et al, 2020), we use a distance field to help capture the relation between the hand and the object. Since we eliminate the requirement of object templates to accommodate more general datasets, we can not define the field using object keypoints (Fan et al. 2023) or templated vertices. Instead, we define the distance field as the vectors between each hand joint and the 10 closest points in the object points. Therefore, we have the distance field as y E 16x300 When the data sample is drawn from HOI datasots, we would also supervisc the distance field calculated from the recovered hand gesture and object shape:
Ldistance = Ily, - 85112
(15)
Besides supervision of the raw MANO parameters for hand, we also supervise the joint position by the loss
Lhand xyx I|LBS(yn) I.BS(
(16)
where LBS(.) is the linear blend skinning process by MANO parametric model to derive the joint kinematic positions. With all the losses introduced, we could train the diffusion model using data with heterogeneous
annotations and for unconditional and conditional generation at the same time. The overall loss is
L DDo (Tunccond Iecond + Lsdistance Ihhand xyz) + Lobj.
(17)

---

Published in Transactions on Machine Learning Research (11/2025)
Table 1: Quantitative evaluation of grasp generation. All models are trained using GRAB and OakInk-shape training set. We evaluate the methods the objects from OakInk-test set, objects generated by JHOD, and objects from the ARCTIC dataset.
Objects
Methods
FID
Pene. Dep.
Intsec.
Sim. Disp. Mean
User Scor.1
GrahNet
26.96
0.70
11.93
3.14
3.80
OakInk-test
GrabNet-Refine
25.26
0.63
4.44
2.78
4.00
JHOD
20.73
0.32
3.87
2.52
4.87
GrabNet-Refine
37.23
1.22
14.20
12.17
1.33
ARCTIC
JHOD
23.92
0.42
4.55
2.91
4.13
GrahNet-
1.17
13.82
10.03
1.33
Self-generated
(HOD(uncond.)
0.51
6.71
7.13
3.53
Table 2: The ablation study about the impact of training data on the generation quality over unseen objects from ARCTIC (Fan et al. 2023). Adding more training data consistently boosts the generation quality on unseon objects.
Oaklnk-Shape
GRAB
Object-only Data
FID +
Pene. Dep. +
Intsec. Vol. +
Sim. Disp. Mean +
User Scor.
27.29
0.51
4.89
3.22
3.87
25.31
0.47
4.61
3.08
4.00
23.92
0.42
4.55
2.91
4.13
Sampling. We follow the canonical progressive denoising process for diffusion models in the sampling stage. To sample for joint (unconditional) HOI generation, we synchronize the schedulers tH = to = Similar to 1022h2022,,,an, at each step 6, we predict a clean sample by &o = (2, (H, from the corrupted code a and then noise it back 2t-1. During sampling for object-conditioned grasp generation, we keep to 0 and Bth # @ = Then the denoiser predicts &n Go(ztg: (t)) and then noise it back to zz-1- In either the unconditional the conditional generation, repeat the process above along T - > until the final 2g is achieved after T itcrations.
4 Experiments
4.1
Setups
Datasets. We combine the data from multiple resources to train the model. GRAB (Taheri et al. (0200) con- tains human full-body poses together with 3D objects. We extract the hands from the full-body annotations. For OakInk (Yang et al. 2022), we use the official training split. for training. We also use the contact-adapted synthetic grasp from the OakInk-Shapp dataset for training. Besides the hand-object interaction data, we also leverage the rich resources of 3D object data to help train the object part in model. The object data is also used to fine-tune the LION prior distribution. Fine-tuning prevents improper object shapes from the original distribution (pre-trained from ShapeNet (Chang et al. 2015)) such sofas, chairs, and bookcases. We combine the objects from GRAB, OakInk (both the objects from OakInk-Image Image with grasp annotation and the objects from OakInk-shape without grasps), Affordpose (Ye et al. 2023b) and DexGraspNet (Wang
et al. 2023) as the object data resource. We also leverage the DeepSDF (Park al.) 2019) as trained Yang al.) 2022) provide synthetic object data and include them in OakInk-Shape.
transform the hands into the object- centric frame. For data. from GRAB (Taheri et al. 2020), extract the MANO parameters from its original SMPL-X (Pavlakos et al. 2019) annotations. Then, we filter out the frames where the right-hand mesh has no contact with the object in training, For the object representation, we randomly sample 2048 points from mesh vertices to represent each object every time we draw the object in both training and inference to avoid overfitting a fixed set of point clouds. For the objects in the OakInk-Shape dataset, we sample the point clouds from the object mesh surfaces uniformly instead of sampling from the vertices because the annotated mesh vertices too sparse on smooth surfaces.
Data Pre- Pre-Processing, We follow the convention in Sakknn--happe
8

---

Published in Transactions on Machine Learning Research (11/2025)
The ablation study about the impact of training data on the generation quality for joint generation of hand and object. Adding more data, either hand-obejet grasp data object-only data, improves the generation quality.
Table 3:
OakInk-Shape
GRAB
Object-only Data L
Penc. Dep. 4
Intsec. Vol. 4
Sim. Disp, Mean 4 User Scor.1
0.92
8.12
8.23
3.33
0.51
6.71
7.13
3.53
0.43
5.55
6.98
3.73
Table 4: Ablation about asynchronous denoising schodulers. We evaluate on the generated object shapes by JHOD.
Asyn.
Pene. Dep. 4 Intsec. Vol.
Sim. Disp. Mean +
0.57
6.52
7.88
0.43
5.55
6.98
Implementation. The DeepSDF in Ink (Yang et al. 2022) is trained per category derive better details and fidelity when interpolating object shapes. We use the DeepSDF to provide synthetic object shapes. For the encoder network to transform modality features to latent codes, always use 2-layer MLP networks with hidden dimension of 1024 and an output dimension of 512. For the denoiser, we follow MDM (Tevet et al. 2022) to use a transformer encoder-only backbone-based diffusion network. After decoding the object latent code by the LION decoder, we derive the point cloud set. For visualization purposes, we conduct surface reconstruction from the generated object point cloud. Though advanced surface roconstruction techniques such some learned solver (Peng al., (021) can provide more details, this part is not our focus and we wish to keep consistency for out-of-domain objects generated. Therefore, we choose the classic Alpha Shape algorithm (Edelsbrunner et al. 1983). Without category labels other constraints, it is challenging to generate object shapes with highly carved details and it is not our focus in this paper.
Baseline Methods There is no commonly adopted benchmark in the area of grasp generation and some similar methods can follow different evaluation protocols. For example, the provided evaluation protocol of G-HOP (Ye et al., 2024) for grasp reconstruction instead of generation and text description or object label is necessary for its generation mode which is not required in our proposed method. On the other hand, another line of works, such UGG (Lu et al. 2023) and DexDiffuer (Weng et al. 2024 focus grasp generation but within the domain of robotic dexterous hand thus there is no trivial way to compare with their in the same setting. In this work, we focus on hand grasp generation with only object shape condition or without any condition. We select the widely adopted method GrabNet (Taheri et al. 2020) as the baseline method to compare with in this section. We also use its refined version GrabNet-Refine in some cerlain comparisons.
Evaluation and Metrics. Unlike the reconstruction tasks where the ground truth is available, it is difficult to do the quantitative evaluation for generation models. We hold the objects from the OakInk-Shape test set and ARCTIC (Fan et al. 2023) dataset for quantitative evaluations. We measure the quality of generated grasp by FID (Frechet Inception Distance) (Heusel et al. 2017) between the images rendered from the ground truth grasps and the generated grasps. We note that we can not directly calculate the FID score on generated 3D parameters as there no commonly used encoder for this purpose and implementing it ourselves can cause many ambiguities. Therefore, we follow the practice Chaan et al. (2022); Gao al. (2022) of computing metrics on rendered images. Also, as the variance of camera views can significantly impact the image distribution, render three views of each GT or generated grasp. We render three views because penetration implausible contact may only be visible from certain angles. This multi-view rendering provide more comprehensive qualitative evaluation but also reduces the potential bias introduced by the setup of single camera view. We also measure the model performance by directly checking the 3D asset quality. We follow previous studies (Yang et al., 2022 2021) to use three metrics: (1) the penetration depth (Penc. Depth) between hand and object mesh (2) the solid interscction volume (Intsec. Vol.) between them and (3) the mean displacement in simulation following (Hasson et al. 2019). Finally, perform user study measure the plausibility and the truthfulness of the generated grasps
9

---

Published in Transactions on Machine Learning Research (11/2025)
binoculars
toothbrush
wineg lass
camera
Figure 3: Generated samples on objects from GRAB (Taheri
2020) dataset.
by scoring (1-5). We invite 15 participants to rate the quality of batches of the mixture of rendering of ground truth grasps and 10 generated grasps. The score 1 indicates "totally fake and implausible" and 5 indicates "plausible enough to be real". Each participant evaluates by averaging the scores over 10 randomly selected batches.
4.2
Doject-Conditionel Grasp Generation
Our method generate visually plausible grasp configuraionss even though the model has not seen these objects during training. We generate samples on the objects from GRAB and OakInk test sets shown in Figure X and Figure We perform the quantitative evaluation on the objects unseen during training with the previously introduced metrics. We follow the previous practice (Yang et al. 2022) to use the widely adopted GrabNet (Taheri et al., 2020) and its refined version GrabNet-refine (Yang et al. 2022) as the bascline models. We train the models on the GRAB and OakInk-shape train splits. The results are shown
in Table D
Compared to the baseline methods, proposed method achieves better generation quality per four metrics: FID, Mesh Penetration Depth, Intersection Volume, and Mean Simulation Displacement. The performance advantages are demonstrated on objects from the OakInk test set, generated by our method or ARCTIC datasot.
OakInk training set and training set contain similar objects though not the same. Such a biased similarity between the training and test sets exists in many HOI datasets and conceals the limitation of generating grasps unscen objects. So we also test the ARCTIC dataset, which is more confidently out-of- distribution from training. GrabNet-Refin Refine fails to generate grasps with decent quality while the advantage of our method becomes more significant. GrabNet-Refine's generation quality is inferior to our method with much larger margin by all metrics. The experiment reveals that some existing methods, such as GrabNet, face difficulty in generalizing to object shapes that are significantly different from training data. On the other hand, our method learns more generalizable and diverse object prior information and more universal grasp generation ability.
4.3 Unconditional Grasp Generation
Our proposed method learns the joint distribution of the latent representation of hand and object. It leverages large-scale object datasets to learn sufficiently generalizable object shape embedding. To measure the quality of unconditional hand-object grasp, the results shown in the last in Table [ The metric
10

---

Published in Transactions on Machine Learning Research (11/2025)
Figure 4: Generated samples on objects from OakInk Yang et al. 2022 dataset.
scores indicate that the unconditional hand grasp generation quality is also good and close to the generation quality on ARCTIC objects. Compared to the object-conditione generated grasps from JHOD, the quality of unconditional generation inferior, which in fact as expected. Here, we also provide the evaluation results by GrabNet-Refine on the same set of objects generated by our mothod for reference. And GrabNet- Refine fails to generalize to this set of unseen objects again just as its failure on ARCTIC objects. Such failure of generalizability is previously under-explored because the training/test splits of HOI dataset usually contain objects from certain categories, with similar scale, geometry, and affordance. The method trained on the training split can learn significantly biased in-domain knowledge about the objects in the test splits. To generate hand grasps on more out-of-domain object shapes would be a challenge to all existing methods.
By designing the method capable of training with object-only data and disentangling the modality represen- tations in the latent diffusion, our method not only has the advantage of generating the object and the hand grasp simultaneously but also achieves much better robustness to generate grasps out-of-domain objects.
4.4 Ablation Study
We now provide ablation studies show the contributions of different resources our method's final performance.
Ablation of grasp generation unseen objects. To provide transparent experimental conclusions, we ablate the training data on the model generation quality. We use the objects from ARCTIC (Fan et al., 2023) to mcasure the conditional generation quality on unseen objects. The quantitative evaluation results are shown in Table 2 Compared to using only OakInk-Shape data for training, adding GRAB data and object shapes from Object-only datasets both improves the generalization qualities. The improvement from extra GRAB data is easy to explain as training with more HO1 data improves the generalizability of the model while it is interesting that adding the object-only datasets also boosts the performance. We believe this is because the extra object data improves the robustness and generalizability of the object encoding. Tt extends the latent representation expressiveness that the diffusion model learns to generate.
Ablation of unconditional generation quality. The ability to generate both hand and object to form a grasp another main ributinu We are interested whether can benefit from the additional training data as well. We present the ablation study in Table 3 Similar to the conditional generation ablation study results, the unconditional generation quality also benefits from additional training data consistently. The metrics of Pene. Dep., Intsec. Vol. and Sim. Disp. Mean indicate that the generated hand-object pair improves plausibility along with adding more training data. The increasing user scoring result suggests that the generated object and hand also become more and more visually realistic along with adding more training data.
Ablation of synchronous denoising schedulers. As one of the main implementation innovations of JHOD, we conduct an ablation about Asynchronous Denoising Schedulers in Table g For the fairness of the comparison, we keep the curriculum of the training the same by mixing data samples from different.
11

---

Published in Transactions on Machine Learning Research (11/2025)
resources. Intuitively, designed this module to decouple the noise level in hand latent and object latent so that the model could better use the data with heterogeneous annotations and learn to denoise certain modality. Without Asynchronous Denoising Schedulers, the noise diffusion and denoising time step of the object part is the same as the hand latent code. It makes the training biased to the HOI dataset where both modalities are available for supervision. Adding Asynchronous Denoising Schedulers, the denoising is learned with more independence for the object part and the hand part. The model can better learn object shape coding to generate hand grasps. Therefore, when evaluating on the out-of-domain objects generated by JHOD, the generation quality is improved as expected.
5 Conclusion
In this work, we propose a joint diffusion model for generating hand-object grasps. By encoding the modalities of objects and hands into unified latent space, model can generate grasps both jointly and conditionally, We address the limitations of existing methods that rely on limited hand-object interaction (HOI) data for training and explore strategies for learning grasp generation that generalizes to a wider variety of objects. Our approach leverages data with partial annotations, thereby mitigating the dependency on fully annotated datasets. The generated grasps, in both unconditional and conditional settings, demonstrate high quality. We believe that our method offers significant advancement in learning generalizable and robust hand-object grasp generation, even with limited fully annotated data.
12

---

Published in Transactions on Machine Learning Research (11/2025)
References
Dexterous hand series. URL http://pt//////aoaoooooooo/ooooa
Adobe. Firefly, 2024. http::/wwww.addddccccc////oooodd
Fan Bao, Shen Nie, Kaiwen Xue, Chongxuan Li, Shi Pu, Yaole Wang, Gang Yue, Yue Cao, Hang Su, and Jun Zhu. One transformer fits all distributions in multi-modal diffusion at scale. arXiv preprint ar arXiv:2303.0653 2023.
Eric R Chan, Connor Z Lin, Matthow A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay, Sameh Khamis, et al. Efficient geomelry-aware 3d generative adversarial networks. In Proceedings of the IEEF/CVF conference computer vision and pottern recog- nition, pp. 16123 16133, 2022.
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprinl Xiv: 1512:112: 2011551.0000 2.0312, 2015.
Sammy Christen, Shreyas Hampali, Fadime Sener, Edoardo Remelli, Tomas Hodan, Eric Sauser, Shugao Ma, and Bugra Tekin. Diffh2o: Diffusion based synthesis of hand-object interactions from textual descriptions. arXiv preprint Xr::::2403. 7827, 2024.
Herbert Edelsbrunner, David Kirkpatrick, and Raimund Seidel. On the shape of a set of points in the planc. IEEE Transactions on information theory, 29(4):551 559, 1983.
Zicong Fan, Omid Taheri, Dimitrios Tzionas, Muhammed Kocabas, Manuel Kaufmann, Michael J Black, and Otmar Hilliges. Arctic: A dataset for dexterous bimanual hand-object manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12943-12952 2023.
Jun Gao, Tianchang Shen, Zian Wang, Wenzheng Chen, Kangxue Yin, Daiqing Li, Or Litany, Zan Gojeie, and Sanja Fidler. Get3d: A generative model of high quality 3d textured shapes learned from images. Advances in neural information processing systems, 35:31841-31854, 2022.
Yana Hasson, Gul Varol, Dimitrios Tzionas, Igor Kalevatykh, Michael Black, Ivan Laptev, and Cordelia Schmid. Learning joint reconstruction of hands and manipulated objects. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 11807-11816 2019.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Aduances in neural infor- mation processing systems, 30, 2017.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXio:2207.12590 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6854. 2020.
Juntao Jian, Xiuping Liu, Manyi Li, Ruizhen Hu, and Jian Liu. Affordposc: A large-scale dataset of hand- object interactions with alfordanceedrriven hand pose. In Proceedings of the IEFE/CV International Conference on Computer Vision, pp. 14713 14724, 2023.
Korrawe Karunratanakul, Jinlong Yang, Yan Zhang, Michael J Black, Krikamol Muandet, and Siyu Tang. Grasping field: Learning implicit representations for human grasps. In 2020 International Conference on 3D Vision (3DV), pp. 333 344. IEEE, 2020.
Jiaxin Lu, Hao Kang, Haoxiang Li, Bo Liu, Yiding Yang, Qixing Huang, and Gang Hua. Ugg: Unified generative grasping. arXiv preprint arXiv:2811.1691 2023.
Zhengyi Luo, Jinkun Cao, Sammy Christen, Alexander Winkler, Kris Kitani, and Weipeng Xu. Grasping diverse objects with simulated humanoids. arxiv preprint ar Xrx:222477 11385, 2024a.
13

---

Published in Transactions on Machine Learning Research (11/2025)
Zhengyi Luo, Jinkun Cao, Sammy Christen, Alexander Winkler, Kris Kitani, and Weipeng Xu. Omnigrasp: Grasping diverse objects with simulated humanoids. Advances in Newral Information Processing Systems. 37:2161 2184, 2024b.
Andrew T Miller and Peter K Allen. Graspit! versatile simulator for robotic grasping. IEEE Roboties & Automation Magazine, 11(4:110--22, 2004.
Supreeth Narasinhaswamy, Uttaran Bhattacharya, Xiang Chen, Ishita Dasgupta, Saayan Mitra, and Minh Hoai. Handiffuser: Text-to-image generation with realistic hand appearances. arXiv preprint
arXiv:243.001693, 2024.
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Decpsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the IEEE/CVF conference computer vision and pattern recognation, pp. 165-174, 2019.
Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmedd A. A. Osman, Dimitrios Trionas, and Michael J. Black. Expressive body capture: 3d hands, face, and body from single image. In Proceedings IEEE Conf. on Computer Vision and pattern Recognition (CVPR), 2019.
Songyou Peng, Chiyu Jiang, Yiyi Liao, Michael Niemeyer, Marc Pollefeys, and Andreas Geiger. Shape points: A differentiable poisson solver. Advances in Neural Information Processing Systems, 34:13032 13044, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint 22x::2204.00125 1(2):3, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEFE/CVF conference. on compuler vision and pattern recognition, pp. 10684 10695, 2022.
Javier Romero, Dimitrios Tzionas, and Michael J Black. Embodied hands: Modeling and capturing hands and bodics together. arXiv preprint 3Xi::2001002611 2022.
Ludan Ruan, Yiyang Ma, Huan Yang, Huiguo He, Bei Liu, Jianlong Fu, Nicholas Jing Yuan, Qin Jin, and Baining Guo. Mm-diffusion: Learning multi-modal diffusion models for joint audio and video generation. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pallern Recognition, pp. 10219 10228, 2023.
Jascha Sick-Dickst,in, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pp. PMLR, 2015.
Omid Taheri, Nima Ghorbani, Michael J Black, and Dimitrios Tzionas. Grab: A dataset of whole-body human grasping of objects. In Computer Vision-ECCV 2020: 16th Europeon Conference, Glasgou, UK, August 23-28 2020, Proceedings, Part IV 16, pp. 581-600. Springer, 2020.
Omid Taheri, Yi Zhou, Dimitrios Tzionas, Yang Zhou, Duygu Coylan, Soren Pirk, and Michael J Black. Grip: Generating interaction poses using spatial cues and latent consistency. In International conference on 3D vision (3DV), 2024.
Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen- Sahen-Or, and Amit H Bermano. Human motion diffusion model. arXiv preprint arX:: 14916, 2022.
Arash Vahdat, Francis Williams, Zan Gojeic, Or Litany, Sanja Fidler, Karsten Kreis, et al. Lion: Latent point diffusion models for 3d shape generation. Advances in Neural Information Processing Systems, 35: 10021 10039, 2022.
Ruicheng Wang, Jialiang Zhang, Jiayi Chen, Yinzhen Xu, Puhao Li, Tengyu Liu, and He Wang. Dexgraspnot: A largo-scale robotic dexterous grasp dataset for general objects based on sinulation. In 2023 IEEE International Conference on Robotics and Aulomation (ICRA), pp. 11359-11366 IEFE, 2023.
14

---

Published in Transactions on Machine Learning Research (11/2025)
Zehang Weng, Haofci Lu, Danica Kragic, and Jens Lundell. Dexdiffuser: Generating dexterous grasps with diffusion models. arXiv preprint arXiv:2402. 02989, 2024.
Sirui Xu, Zhengyuan Li, Yu-Xiong Wang, and Liang- Yan Gui. Interdiff: Generating 3d human-object interactions with physics-informed diffusion. In Proceedings of the IEFE/CVF International Conference Computer Vision, pp. 14928-1494) 2023.
Lixin Yang, Xinyu Zhan, Kailin Li, Wenqiang Xu, Jiefeng Li, and Cewu Lu. Cpf: Learning a contact potential field to model the hand-object interaction. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 11097-11 1199-111066 2021.
Lixin Yang, Kailin Li, Xinyu Zhan, Fei Wu, Anran Xu, Liu Liu, and Cewu Lu. Oakink: A large-scale knowl- edge repository for understanding hand-object interaction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Paftern Recognition, pp. 20953-20962, 2022.
Yufei Ye, Poorvi Hebbar, Abhinav Gupta, and Shubham Tulsiani. crffusion-guider reconstruction of every- day hand-object interaction clips. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 19717-19728, 2023a.
Yufei Ye, Xueting Li, Abhinav Gupta, Shalini De Mello, Stan Birchfield, Jiaming Song, Shubham Tul- siani, and Sifei Liu. Affordance diffusion: Synthesizing hand-object interactions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22479 22489, 2023b.
Yufei Ye, Abhinav Gupta, Kris Kitani, and Shubham Tulsiani. G-hop: Generative hand-object prior for interaction reconstruction and grasp synthesis. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1911-1920, 2024.
Hui Zhang, Sammy Christen, Zicong Fan, Otmar Hilliges, and Jie Song. Graspxl: Generating grasping motions for diverse objects at scale. arXiv preprint 20x 2024a.
Mengqi Zhang, Yang Fu, Zheng Ding, Sifei Liu, Zhuowen Tu, and Xiaolong Wang. Hoidifiusion: Generating realistic 3d hand-object interaction data. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8521-8531, 2024b.
Keyang Zhou, Bharat Lal Bhatnagar, Jan Eric Lenssen, and Gerard Pons-Moll. Gears: Local geometry-aware hand-object interaction syuthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition, pp. 20634 20634-20643, 2024.
15

---

Published in Transactions on Machine Learning Research (11/2025)
Figure 5: Samples of the unconditional generation of hand grasp together with objects.
A Grasp Diversity
Given single object shape, we could generate set of different grasps onlo it. Though diversity expected to be constrained by the hand grasp data available in training, there are some other implicit but more fundamental cues to help generate diverse hand grasp. This underlying but fundamental constraint be leveraged to enhance the generation robustness over unseen objects if the feature of the object shape is sufficiently generalizable. For example, the model can learn to avoid penetration between the object volume and the hand shape and certain fingers should be close to the surface of objects to form a physically valid grasp. We generate grasps with unseen objects from the OakInk-shape test set as the condition in Figure [ We observe some grasp patterns not provided in the training set, for example, holding a dagger between two fingers. As the implicit concept of forming a valid grasp is always conditioned to object shapes, strategy of exposing the model training to more diverse object shapes help to learn grasp patterns unseen and even out-of-domain objects. We believe this is a key reason for making our method outstanding when generating grasps on out-of-domain objects.
B Dataset Statistics
Here we provide more details about the statistics of the datasets involved in our training and evaluation in Table The HOI datasets with deformable and articulable hand models, ie., MANO (Romero et al. 2022), face severe limitations of object resources. Combining the training set of GRAB (Taheri al. 2020 and OakInk-Image (Yang al. (2022) datasets still make just ~100 objects. On the other hand, Afford- Pose (Tian et al. 2023) and DexGraspNet (Wang et al.) 2023) contain more object shapes but the different choices of hand models make them hard to integrate with MANO-paramest parameterized datasets. Fortunately, we have the large-scale 3D object shape dataset ShapeNet (Chang et al. 2015) with more than 50,000 objects. We combine the objects from these datasets in the training for the object part. They help to construct a more universal prior distribution for object generation and the grasp posterior distribution.
C Grasp-conditioned Image Generation
We demonstrate using JHOD as a creativity tool to generate 3D grasps and the corresponding images. In Figure 7 we first use JHOD sample interaction pairs of hands and objects. With the rendering of the 3D hand-object grasp as the input, we applied Adobe Firefly (Adobel 2024). an existing tool generate the images with the depth and edge reference of the 3D pairs. The hand pose and the geometry
16

---

Published in Transactions on Machine Learning Research (11/2025)
Figure 6: Given a single object, our method is capable of generating different grasps on it.
17

---

Published in Transactions on Machine Learning Research (11/2025)
Table 5: Statistics of the datasets used for training. ShapeNet contains the most object assets. However it is generic object shape dataset, thus many included objects are not proper for hand grasp. OakInk-Shapo contains rich grasp and object data but not provide annotation for object transformation. We use the grasp data from GRAB and the object data from Pose as the supplement during training.
Datasets
#obj
#grasp
real /syn.
hand model
ShapeNet
51,300
real
GRAB (ahheii et al. 2020
51
1.3k
real real real + synthetic
MANO
Romero et al.
2022
OakInk- Image Yang et al.
2022
100
49k
MANO
Romero et al.
2022
DakInk-Shape Yang et. al.
2022
1,700
MANO
Romero et, al
2022
AffordPose (Jian ct al. 2023
641
26k
synthetic
GraspIt
Miller & Aleen
2004
DexGraspNet (Wang al.
2023)
5,355
1.32M
synthetic
Shadow (sha
Holding saki bottle Holding
Holding snall
Holding big bottle
Holding milk box
Figure 7: Samples of images generated by Adobe Firefly with synthesized hand grasp by our proposed method as the condition.
provide guidance for Firefly to complement the details given simple text prompts. Figure S compared grasping images authorized purely by text prompts, images guided by JHOD's output have more consistent gestures, better-aligned object geometry and boundaries, and fewer hand artifacts. Therefore, with the rise of text-to-image generation, our hand-object grasp generation model can serve proxy to enhance the consistency among multiple instances and the visual plausibilities. We also provide closer look at their detailed in Figure g to compare the results with and without the generated grasp as a proxy. At each row, the images are generated from the same text prompt as shown at the bottom. There are two main benefits of generating images conditioned to the grasp. It first allows a set of images with aligned hand and object geometry and boundaries which can be useful for image and video editing. On the other hand, even though the Adobe Firefly generator has shown a significant advantage over the public Stable Diffusion in eliminating artifacts, still observed many finger artifacts when generating the images without grasp condition. Fortunately, with the generated grasp, including the object shape and the hand, as the condition, the image generator can produce significantly fewer artifacts, especially artifact fingers, which have been a notorious issue in image generation recently. We provided some zoomed-in examples at the bottom. Moreover, compared to previous works using hand shapes to guide image generation (Ye et al. 2023b Narasimhaswamy et al.) 2024), we could generate both the object and the hand. Therefore, we could use the whole scene of hand-object interaction as the condition and thus enable more controllable details in the generated image.
18

---

Published in Transactions on Machine Learning Research (11/2025)
Not Grasp-conditioned
Grasp-conditioned
Prompt:
Holding
Prompt:
Pouring soda from battle
Figure 8: Using the same text-prompted image generation tool, we synthesize photo-realistic images with and without the grasp as the geometry condition. The grasp-conditioned image generation can have broad applications in image and video editing.
19

---

Published in Transactions on Machine Learning Research (11/2025)
Not Grasp-conditioned
Grasp-conditioned
Ho Holding milk box
Holding saki bottle
Receive huge pencil
Ho lding Whiskey flask
Ho Holding small vase
Hand vegetable
not grasp-conditioned images with artifact fingers
Figure 9: More examples of generating images of grasps with and without the condition from our generated
grasps by the same image generation tool. Without plausible grasp as the condition, there are more frequent unrealistic artifacts in the generated images, especially the number and pose of fingers. We provide zoomed-in bad examples at the bottom.
20

---

Published in Transactions on Machine Learning Research (11/2025)
no contact
unplausible pase
nesh penetration
Figure 10: Bad samples of conditional grasp generation given objects from GRAB (gray) and OakInk-Shape (green).
D Failure cases
To provide transparent evaluation of our proposed method, we still notice some failure cases the unseen objects from OakInk and GRAB datasets shown in Figure 10 There are in general three patterns of failures: (1) the hand and the object have no contact, making the grasp physically implausible; (2) the hand skin is twisted or the pose is not able to grasp the object in human common sense; (3) the mesh of object and hand has penetration and intersection. There can be some reasons that make these happen. First all, we do not explicitly supervise the contact between the object surface and hand as this is a rare annotation on many data sources, and calculating can be computationally expensive. On the other hand, some implausible grasps have pretty good visual quality as shown in the middle figure. However, according to our life experience, manipulate the object. This has gone beyond our scope in this work supervision such as the physical demonstration in physics simulator. Finally, we use the object point cloud to represent object shapes to allow grasp generation on universal objects without a template, this causes the potential penetration between object mesh and hand mesh as the object mesh ambiguous and unknown our method. Despite the failure cases, we note that they make only a very small portion of the generated samples (less than 10% by rough estimation). We show them here for transparency and help discussion about future works in this area.
columns in the know that the grasp is not physically plausible to do not have any physics-aware
21

---

Published in Transactions on Machine Learning Research (11/2025)
E
Hand Contact with Object
Hand contact with the object is a key component of valid hand-object grasp. In Figure 11 we visualize the contact area on the hand mesh when the hand is posed lo grasp an object. We provide the visualization of the hand-object grasp in three different camera views to showcase the full configuration of the grasp, and also a hand-only visualization to more clearly justify the contact area (in red). Following existing practice while making a stricter standard justify valid "contact", we the distance threshold between the hand mesh face and the object mesh face to be 0.005 meter for the visualization.
F Qualitative Comparison
In addition the quantitative results presented in the experiment sections, we now provide qualitative comparisons between method and the baseline GrabNet (Taheri et al. 2020), shown in Figure 12 We randomly sample objects from the "bottle" category in OakInk that were not seen during training. Grasps are generated for each object without any cherry-picking. Tn each result pair, the left-hand grasp is produced by our method, while the right-hand grasp is from GrabNet. As shown, GrabNet generates grasps that penetrate the object in pairs (1), (2), (4), and (6), and lacks plausible contact pairs (3) and (8). In contrast, our method produces more realistic grasps most cases, although some penetration is still observed in pairs (2) and (6). Overall, these comparisons suggest that our method achieves superior qualitative performance grasp generation.
22

---

Published in Transactions on Machine Learning Research (11/2025)
Figure 11: Visualization of the generated hand-object grasp and the corresponding hand contact area. Here, we set the distance threshold to justify the contact between a hand mesh face and an object mesh face be 0.005 meters. The contact area is rendered in red.
23

---

Published in Transactions on Machine Learning Research (11/2025)
(1)
(2)
(3)
(4)
(5)
(6)
(8)
Figure 12: Grasps generated by our proposed method and GrabNet. (Taheii et al. 2020). In each pair of grasps, the left-hand grasp generated by our method and the right-hand one is generated by GrabNet.
24


Reviews:
[{"review_id": "bLQQKx5Wng", "reviewer": "Reviewer_wCFg", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "In general, as someone who would use grasp-pose generation downstream in robotics in particular, I have lower trust in evaluation metrics that do not use these downstream use cases. As mentioned in Section 4.1, it can be difficulty to do quantitative evaluation of hand grasps when we do not have a single ground truth like we do for reconstruction losses. The measures of penetration depth, intersection volume, etc. are good, but the paper would be stronger if there were some qualitative comparisons between generations from JHOD and GrabNet. I believe all the images are only from JHOD and it would be nice to see commentary on general trends that differ betwee JHOD and GrabNet, with examples.\n\nAside from this, the way diffusion is done is well-founded and is what you would generally do when training a multimodal diffusion model. I believe the paper is solid enough.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors aim to generate hand-object grasps by diffusion. The authors argue that prior datasets for this are sometimes limited in the class of objects, or have different models for the hand geometry, causing poorer generations on objects that are out of distribution.\n\nTo handle this, the authors design a joint denoising diffusion model operating in both object-space and hand-space. Each modality (object, hand) is encoded into a latent $z_O, z_H$, which is concatenated into a longer latent $z$. Denoising happens over this $z$, which allows the denoising to process the object and hand modalities together. The denoised latent is then decoded into a generated object and hand.\n\nSuch a model is used in two ways: to generate samples of ${O, H}$ unconditionally, or to generate hand samples $H$ conditioned on a fixed object ${O}$. The latter is what we are most interested in for downstream use cases.\n\nThe object encoder-decoder is a frozen model, from the LION paper (Vahdat et al. 2022), which was trained exclusively on 3d object generation from a large set of objects, so the paper exclusively trains the hand generation, using the narrow hand-object datasets and relying on generalization from LION to handle newer objects. To train this, the paper proposes an asynchronous denoising schedule, which applies two different noise schedules to the object embedding and the hand embedding. This is done because the model will be called either with a fully noised object and hand, or a defined object and noised hand, and the denoiser needs to handle both these cases.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please update the references to wrap them in (parentheses) rather than in-lining all the author names. Please include qualitative examples of generated grasps from baseline methods.", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "IAQpl7e5bs", "reviewer": "Reviewer_vwUU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- This paper introduces an interesting way to leverage the existing heterogeneous 3D datasets, including the hand-object grasp dataset, the object shape dataset, and the synthesized datasets. \n- While overall design choices are fairly well known and simple, the idea of leveraging external datasets and model (e.g., LION) for overcoming the scarcity of the data in the field is interesting. Specifically, the authors' effort to avoid distinguishing object classes when training the diffusion model is interesting: they attempted to encode object shapes based solely on the objects' geometric cues. Although they used the pre-trained object encoder-decoder for this, this reviewer thinks that it's better than existing approaches, which typically rely only on limited hand-object grasp datasets.\n\n**Weaknesses**\n- This reviewer has concerns regarding the evaluation part. \n- First, the authors propose to measure the FID score from the rendered images of the 3D hand and objects. While this reviewer agrees that there are no outstanding metrics to measure such a generation task, this reviewer thinks that the FID of rendered 3D objects is not so robust. Specifically, when measuring the feasibility or realism of the hand-object grasp, the camera views for rendering the 3D scene would be critical factors that may perturb the FID scores. How did the authors select the views for rendering the scene? More details, analyses, and justifications would be appreciated, e.g., providing some sample images to measure FID (both GT and generated sets), investigating the influence of the number of views or view angles on the FID score, etc.\n- Next, the provided qualitative samples are limited. The samples presented in Figures 3, 4, and 5,6 are heavily overlapped. Moreover, it would be more convincing if the authors show more qualitative samples for the object-conditioned grasp generation on the unseen or unusual object shapes.\n- It would be good if the authors could also visualize contact or affordance information for the generated hand-object grasp samples. While the authors report the quantitative results and the quantitative ablation study for it, this reviewer thinks it would be more compelling if the authors could visualize the contact or affordance, especially for the ablation study (Table 3). Since the metrics for penetration, intersection, etc, have improved when using more datasets (in Table 3), this reviewer thinks the qualitative results for contact points would also be improved.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a diffusion model for joint Hand-Object grasp generation (both conditional, unconditional). The proposed Joint Hand-Object Diffusion (JHOD) model is trained with the combination of 1) existing hand-object interaction datasets, 2) object shape datasets, and 3) synthetic object datasets, and the proposed diffusion training scheme is designed to handle such heterogeneous supervision from partial annotations. Given such a mixed dataset, the authors train the latent diffusion model (JHOD) denoiser network, hand encoder, and decoder jointly, while the object encoder and decoder are borrowed and fixed from the existing work, LION. The performance is evaluated using FID scores of mesh renderings, and some feasibility terms for checking hand-object penetration, and finally, the user preference scores.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Overall, this reviewer found the paper interesting and sound. This reviewer expects the authors to address the concerns about the evaluation side, especially with 1) more analyses and justifications on using FID, 2) more qualitative samples, and 3) visualizing the contact or affordance of the generated hand-object grasp scenes.", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "jKLLDF6NGk", "reviewer": "Reviewer_TZMo", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n1. Unified Latent Diffusion Framework: The proposed model can handle both unconditional and conditional generation, which is a significant step forward in general-purpose hand-object interaction modeling.\n\n2. Extensive Experiments: The paper evaluates on multiple datasets (GRAB, OakInk, ARCTIC), includes detailed ablations (data sources, scheduler design), and uses a range of metrics (FID, penetration depth, intersection volume, user study scores).\n\n3. Generality & Practical Utility: The method supports multiple downstream applications such as photorealistic image synthesis, without requiring language prompts or category labels.\n\n**Weaknesses:**\n\n1. Motivation for Using Object-Only Data Is Underexplained: While the idea of leveraging large-scale object-only datasets to improve hand-object grasp generation is compelling and shows empirical gains (e.g., in Table 2 and 3), the paper lacks a clear theoretical or intuitive justification for why object shape reconstruction via diffusion contributes to grasp generation, especially in the absence of grasp-specific supervision.\n\n2. Joint Latent Code Denoising Ambiguity: The proposed asynchronous denoising schedule enables training on heterogeneous data, including object-only samples by filling the hand latent with Gaussian noise. However, it is unclear whether this training setup introduces distributional mismatch or learning conflicts between object-only generation and unconditional/conditional generation.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces Joint Hand-Object Diffusion (JHOD), a novel latent diffusion model for generating hand-object grasp configurations in both unconditional and object-conditioned settings. Unlike previous works that rely on limited hand-object interaction datasets, JHOD leverages large-scale object-only datasets to learn more robust object embeddings and jointly models hand and object representations in a unified latent space. A key technical contribution is the use of asynchronous denoising schedulers that allow training on heterogeneous data (e.g., object-only or grasp-only), which boosts generalization and flexibility. The authors conduct thorough evaluations and ablations, showing strong generalization to unseen and out-of-domain objects, outperforming GrabNet and GrabNet-Refine by a notable margin. Additionally, the model enables creative applications such as grasp-conditioned photorealistic image generation.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Could you elaborate on the mechanism by which training the object reconstruction (via diffusion on object-only data) helps improve the grasp generation quality? Is it purely through improved object embeddings or is there implicit transfer to hand latent decoding?\n\n2. In the training stage with object-only data, the hand part of the joint latent code is sampled from noise. How does the model avoid learning a distribution that conflicts with the objectives of conditional/unconditional grasp generation, where the hand code is expected to be meaningful?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)