TOUCH: Text-guided Controllable Generation of Free-Form Hand-Object Interactions
Authors: Guangyi Han
Wei Zhai
Yuhang Yang
Yang Cao
Zheng-Jun Zha
Venue: ICLR 2026 Poster (2026)
Keywords: hand-object interaction
3D generation
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We extend hand-object interaction generation beyond simple grasping to a diverse range of controllable manners, such as pushing, poking, twisting, and rolling.

Abstract: Hand-object interaction (HOI) is fundamental for humans to express intent. Existing HOI generation research is predominantly confined to fixed grasping patterns, where control is tied to physical priors such as force closure or generic intent instructions, even when expressed through elaborate language. Such an overly general conditioning imposes a strong inductive bias for stable grasps, thus failing to capture the diversity of daily HOI. To address these limitations, we introduce $\textbf{Free-Form HOI Generation}$, which aims to generate controllable, diverse, and physically plausible HOI conditioned on fine-grained intent, extending HOI from grasping to free-form interactions, like pushing, poking, and rotating. To support this task, we construct $\textbf{WildO2}$, an in-the-wild diverse 3D HOI dataset, which includes diverse HOI derived from internet videos. Specifically, it contains 4.4k unique interactions across 92 intents and 403 object categories, each with detailed semantic annotations. Building on this dataset, we propose $\textbf{TOUCH}$, a three-stage framework centered on a multi-level diffusion model that facilitates fine-grained semantic control to generate versatile hand poses beyond grasping priors. This process leverages explicit contact modeling for conditioning and is subsequently refined with contact consistency and physical constraints to ensure realism. Comprehensive experiments demonstrate our method's ability to generate controllable, diverse, and physically plausible hand interactions representative of daily activities.

Published as a conference paper at ICLR 2026
TOUCH: TEXT-GUIDED CONTROLLABLE GENERA- TION OF FREE-FORM HAND-OBJECT INTERACTIONS
Guangyi Han Wei Zhai' Yuhang Yang', Yang Cao', Zheng-Jun Zha
MoE Key Laboratory of Brain-inspired Intelligent Perception and Cognition,
University of Science and Technology of China (hanguangyi @mail., wzhai0560,
yyuhangemail., forreste, zhazj8)ustc.edu
ABSTRACT
Hand-object interaction (HOI) is fundamental for humans to express intent. Exist- ing HOI generation research is predominantly confined to fixed grasping patterns, where control is tied to physical priors such as force closure or generic intent in- structions, even when expressed through elaborate language. Such an overly gen- eral conditioning imposes a strong inductive bias for stable grasps, thus failing to capture the rich diversity of daily HOI. To address these limitations, we introduce the new task of Free-Form HOI generation, which aims to generate controllable, diverse, and physically plausible HOI conditioned fine-graines intent, includ- ing non grasping actions like pushing, poking, and rotating. To support this task, we construct WildO2, the first large-scale, in-the-wild 3D HOI dataset, which in- cludes non-grasping motions derived from internet videos; it contains 4.4k unique interactions across 92 intents and 610 object categories, each with detailed seman- tic annotations. Building on this rich dataset, we propose TOUCH, a three-stage framework centered on a multi-level diffusion model that facilitates fine-grained semantic control to generate versatile hand poses beyond grasping priors. This process leverages explicit contact modeling for conditioning and is subsequently refined with contact consistency and physical constraints to ensure realism. Com- prehensive experiments demonstrate our method's ss ability to generate controllable, diverse, and physically plausible hand interactions representative of daily activi- ties. Project page is https::///// /etee://guannicnne wwtpiggiidigggiiiio github. io oto.ooooiiiiioe htt.hhu.ooooooo1tttutut..
INTRODUCTION
Mojee- Interaction (HOI) is fundamental to expressing intent and executing tasks in human daily life. and the ability to generate controllable interactions is crucial for AR/VR, robotics, and embodied AI (Zheng et al. 2025). While existing HOI generation research has progressed from ensuring physical plausibility (Fang al. 2020) to incorporating semantic controllability (Li al., 2024b Yang et al. 2023), its scope remains predominantly confined to a grasp-centric paradigm. The control signals these methods, whether simple physical constraints like force closure or coarse high-level instructions (e.g., verb-noun noun pairs), are often overly general. This simplified conditioning imposes strong inductive bias that primarily favors the generation of stable grasps (Taheri et al., 2020), sacrificing interaction diversity. Furthermore, even with more sophisticated control such as detailed natural language (via LLMs) (Zhang et al., (2025a (). the underlying model designs and in- herent inductive biases are still fundamentally geared towards generating only grasping interactions, driven by historical focus and prevailing representations. Consequently, these approaches lack the fine-grained control and inherent capability to capture the diverse non-grasping interactions found in the real world, including varied hand poses, contact details, and nuanced semantic intent.
To bridge the gap between the limited scope of current methods and the complexity of real-worle interactions, we introduce the task of Free-Form HOI Generation. The goal is to break grasp- centric limitations and shift towards generating diverse interactions, including the vast array of non- grasping manipulations. This task emphasizes expressiveness and controllability in the generation
Equal contribution. * Corresponding author.

---

Published as a conference paper at ICLR 2026
Laboratory Collection
Daily Hand- Object Interaction
Free-Form Contacts
Free-Form Actions
Condition
Diverse Objects
Tip water bottle over.
SSCs:
Object,
Dataset
Affordance,
HOI4D,
generall: tipping [obj. categary] physical: Apply (hand contact part] to exert controlled
Text....
drive
GRAB,
Oakink,
on the [obj, of [ob], categoryl. causing
part]
tip over.
hand contact part: [index pad]
Grasp
obj contact part: cap
Generation:
obb category: water bottle
DSCs
Figure I: Overview. We extend HOI generation beyond laboratory "grasp" settings (left) toward broader daily HOI modalities (right), enabling the modeling of more human-like interactions. Our dataset WildO2, built from Internet videos, covers more contacts, more objects, and more actions, and is enriched with descriptive synthetic captions (DSCs) to support fine-grained semantic control- lable HOI generation with our method, TOUCH.
process, aiming to synthesize interactions that are not only physically plausible but also semantically rich and truly adaptable to complex human intentions.
The core challenge of this task lies in two aspects: what to generate and how to generate it. The former pertains to spatial plausibility: the model must break free from restrictive grasping priors (e.g.. palm position and orientation, contact region assumption (Ye et al.) 2024: Jiang et al.] (22T)) to explore vast yet physically valid interaction space. To address this, we propose that contact relationships serve as a powerful cue to constrain this high-dimensional space, offering a more nu- anced understanding of physically valid interactions. The latter pertains to semantic controllability: the model must accurately map fine- ime-grained textual instructions to specific hand configurations and contact regions. The prior knowledge within Large Language Models (LLMs) offers promising pathway for this guidance (Tang et al. 2023). A major obstacle to learning this complex mapping is the lack of 3D training data for diverse daily interactions, as existing datasets (Zhan et al. 20244 Fu al. (025) are mostly limited to lab-based grasping and object instances. Hardware and capture challenges make large-scale real-world 3D data collection difficult. In contrast, abundant 2D HOI videos online provide rich and realistic daily interaction behaviors.
To tackle the proposed task and challenges, we present TOUCH, a three-stage framework for con- trollable free-form HOI generation. First, we explicitly model the contact on the surfaces of the hand and the object separately by jointly encoding spatial point-cloud relations and semantic information, providing strong spatial priors mitigate uncertainty from the high degrees of freedom in inter- action position and pose. We further incorporate part-level hand modeling for more precise action control. Second, we employ multi-level diffusion model with attention-based fusion of semantics and geometry: coarse- sousee-rraied intent and global object geometry guide the early diffusion stages, while fine-grained text and local contact features refine detailed motions in deeper stages, enabling fine-gne fine-grained semantic controllability. Finally, we introduce self-supervised contact consistency and physical plausibility constraints to optimize the generated interactions, ensuring realism and physical feasibility. Compared to prior methods restricted to grasp generation, TOUCH naturally generalizes to diverse free-form HOI such as pushing, pressing, and rotating.
Additionally. based on 3D object reconstruction (Xu et al., 2024), we introduce an automated pipeline to build the dataset WildO2 that jointly recovers and optimizes high-quality 3D hand-object interaction samples from internet videos annotated with interaction intent. By leveraging vision- language models (Bai et al. 2023b), we generate fine-gre grained semantic annotations, resulting in the 3D daily HOI dataset covering diverse interaction intents.
Our main contributions are: (1) We propose to extend the HOI from constrained grasping to a broader, more realistic, and more diverse set of daily interactions. (2) We propose TOUCH, new
2

---

Published as a conference paper at ICLR 2026
framework that generate natural, physically reasonable, and diverse free-form HOI under fine- grained text guidance. (3) We build an automated pipeline and construct WildO2, an in-the-wild 3D dataset for daily HOI, providing critical resource that enables future research in this domain. Extensive experiments demonstrate the superiority of TOUCH.
2 RELATED WORK
2.1 HAND-OBJECT INTERACTION DATASETS.
Existing 3D hand-object interaction (HOI) datasets are predominantly collected in controlled labo- ratory settings, relying either on physics-based -based simulation synthesis (Hasson et al. 2019) or motion capture systems to record real interactions (Hampali et al. [2000] Liu et al. [022] Yang et al. 2022
Brahmmbatt et al. 2020). Although these datasets provide valuable support for modeling 3D HOI, they suffer from limited diversity due to constrained camera setups, small number of participants, and a restricted set object instances. In contrast, large-scale in- in-the-wild video datasets (Damen et al. 2020; Grauman al. 2022, Shan et al., 2020) contain abundant HOI clips, but lack high- quality 3D annotations. Some studies have attempted to annotate subsets of these videos in 3D using object template-based optimization methods (Cao et al. 2021 Patel et al., [2022;; however, due to the high diversity of open-set objects, scaling such approaches remains challenging.
2.2 TEMPLATE-FREE HOI RECONSTRUCTION
The core bottleneck in reconstructing HOIs in the wild has long been the recovery of diverse ob- ject geometries. While existing template-free approaches (Fan al.] 2024 Ye al. 2022) avoid predefined object model constraints, they are typically trained on limited datasets and exhibit poor generalization to novel objects. In recent years, multi-view diffusion models (Liu et al., 2023a) and
large-scale reconstruction models (LRMs) (Hong al. (024) have enabled high-quality 3D mesh reconstruction directly from single images (Xu et al. 2024; Liu et al. 2024b) or text prompts (Poole et al. 2022), demonstrating strong generalization capabilities. Motivated by these advances, eral HOI studies have explored image-to-3D reconstruction pipelines to handle open-set objects in the wild. However, due severe hand occlusion, these methods often rely image inpainting to complete occluded regions (Tian et al. 2025 Liu et al.) [2024a] Wen et al. 20255 Liu et al. (2024c), employ text-to-3D generation align with coarse reconstruction results (Wu al.] [2224] Chen et al. 2025). Nonetheless, most of these pipelines depend on heuristic completion or registration strategies, resulting limited geometric consistency with the input, and have yet be validated at scale in an automated manner.
2.3 DATA-DRIVEN CONTROLLABLE HOI GENERATION.
In the evolution of HOI generation, interaction guidance has progressively advanced: from coarse control based on grasp type (Feix et al. 2013), to object-conditioned generation (Karunratanakull et al. 2020; Diang al. 2021), and further to task/action-level intent constraints (Christen et al.; 2024 Yang et al.) 2024b a Yu et al.; (025). To enhance physical plausibility, contact penetration loss and hand anatomical constraints have been widely adopted (Wei et al., 2024). Additionally, explicit modeling of hand part segmentation and contact relationships with objects has been shown to improve physical realism and detailed expression of interactions (Liu al. 2023b Zhang al., 2024 Li et al. 2024a). Building on these efforts, we propose a multi-level controllable generation framework trained on our newly constructed daily HOI dataset, enabling finer- finer-grained semantic tent control and the flexible generation of free-form HOIs that align with complex human intentions.
DATASET
3.1 DATA COLLECTION AND PROCESSING
Our goal is construct diverse dataset of 3D hand-object interactions from wn-the-wild videos. A primary challenge in this process is the severe occlusion of the object by the hand, which com- promises the quality of 3D object reconstruction. To address this, introduce semi-automated
3

---

Published as a conference paper at ICLR 2026
Image Collection & Processing
H&O Reconstruction
Contact Alignment Optimization
Hand Reconstruction
M
hol
Contact
H
hot
ppyy
Impairet
One-image-to--
Obj-only rref
Co Gy
Leam
Vol
afffine
SSCs
feature
MP
Apply [thumb pad, index pad, ring nail] to gently lift one end of the [edge]] [plate] and then release it to drop down.
DSCs:
Figure 2: The proposed data reconstruction and annotation pipeline for WildO2. The process begins with O2HOI frame pair extraction from in- the-wild videos, followed by three-stage pipeline for 3D reconstruction, camera alignment, and hand-object refinement that produces high-fidelity interaction data.
data reconstruction and annotation pipeline, centered around a novel Object-only to Hand-Object Interaction (O2HOI) frame pairing strategy.
We begin by filtering the Something-Something V2 dataset (Goyal et al. 2017), which is rich in goal-directed human actions, obtain 8k single-hand, single-object interaction clips. For each clip, automatically extract an O2HOI pair (details in Appendix): an object-only frame Leefs where object is unoccluded, and corresponding interaction frame Inoi- obtain a complete ob- ject mask in the interaction frame, segment the object in Ire/ using SAM2 (Ravi et al., 2024) and then transfer this mask to Inoi via robust dense matching model (Edstedt et al. (2024) yield- ing Minpaint- This mask transfer strategy offers distinct advantage avoids the geometric inconsistencies of diffusion-basee inpainting (Liu et al. 2024a) while being significantly scalable than manual completion (Wen al.) 2025). Consequently, approach facilitates the automated, large-scale generation of high-fidelity 3D assets for reconstruction.
common ternatives:
3.2 DATA RECONSTRUCTION PIPELINE
Based on the O2HOI pairs, we build a three-stage generation pipeline to recover 3D HOI.
Stage Initialization. For each pair, reconstruct textured object mesh Vo Vecooa from the object-only frame I using an image-to-3D model (Xu et al. 2024). Concurrently, we estimate initial MANO (Romero et al., 2017) hand parameters Hinit from the interaction frame Inos using state-of-the-arr hand reconstruction method (Pavlakos et al. 2024).
Stage 2: Camera Alignment. A challenge arises from coordinate system misalignment: the object recon is created in canonical space of the object-only frame Tref. while the hand exists in the camera space of the interaction frame Ihoi- To unify them, we align VO recon to an object-centric global coordinate system relative to the interaction frame by optimizing the camera projection matrix K and extrinsics (R, t). This is achieved by minimizing a camera alignment loss, Leams via differ- entiable rendering. The optimization proceeds in two phases: we initially use mask IoU, Sinkhorn (Cuturi 2013) loss, and an edge penalty term (to prevent the object from moving out of view). Once the loU surpasses a threshold, we introduce scale-invariant depth (Eigen et al. 2014) and RGB reconstruction losses for fine-tuning. The overall objective is formulated as:
mesh Vo
min Leaam Lmmaskc Lsinkhorn K.R.t
Ledgs + Xine (ndephh + Ligh).
Stage 3: Hand-Object Refinement. With the aligned camera and object, we refine the initial hand parameters Hini to achieve physically plausible contact. Specifically, we cast rays from the camera center through pixels within the interaction mask Mmpgsiut- The intersection points of these rays with the 3D hand and object geometries define potential 3D contact zone. We then optimize H using refinement objective Latign, which combines 2D evidence with 3D physical constraints: hand

---

Published as a conference paper at ICLR 2026
-Others
intercttiiie fallure
TD mm Indecpad
Pone Estimation Fallure
3%
9%
Hand Part Contact Distribution
31%
55%
nonts
(a)
(b)
Figure 3: Dataset Statistics: (a) Breakdown of WildO2 reconstruction outcomes. (b) An illustration of the interplay between the most frequent object categories, interaction types, and hand contact regions. Object and action definitions are adapted and refined from (Goyal et al. (017). Contact regions are derived based on our dataset analysis. (c) Specific segmentation of the 17 hand parts and their contact frequency distribution in the dataset, along with a contact heatmap of the entire hand.
mask loU k) 2D joint reprojection error (Lyza), an ICP loss on the 3D contact zone (Liep).
and physical constraints for contact, penetration, and anatomy based on (Yang et al. 2021).
min; Lutign H
H + Lypa + Liep + Lphy: Lphy = Leontact + Lpene + Lanatomy + Leetf- (2)
This pipeline yields 4,414 high-quality 3D hand-object interaction samples after a final stage of manual inspection and refinement, which constitute the ground truth of our dataset.
3.3 DATA ANNOTATION AND STATISTICS
We enrich our dataset with a multi-level annotation system, generating over 44k annotations. A statistical overview is provided in Fig. 3
with further details in the Appendix.
3D Geometry and Transformation. Each sample includes the final hand-object meshes (V Hr, Vo) and the corresponding camera parameters derived from our generation pipeline. Contact Maps. We compute dense contact maps between the hand and object surfaces. To handle varying object scales, our method robustly identifies contact regions by combining relative and absolute distance thresholds with bidirectional nearest filtering. Multi-Level Language Descriptions. We provide two levels of textual deseriptions. We inherit the template-basse Short Synthetic Captions (SSCs) from Vision-Language Model (VLM) (Bai et al. 2023b) to generate more detailed Descriptive Synthetic Captions (DSCs), which manually verified for quality and relevance. Fine-Grained Hand Part Segmentation. We segment the hand mesh into 17 parts, including finger pads, nails, knuckles, palmar, and the dorsal region. This partitioning scheme goes beyond the coarse divisions commonly used in grasp generation tasks (Hasson et al., 2019) Liu et al. (023b) -hich often focus only on contact on the inner hand hand-by also accounting for contact on the dorsal side. This fine- -raiindd segmentation supports detailed local interaction analysis and facilitates alignment with the semantic descriptions in the DSCs.
V2 (e.g.. "picking [Something] up")) Additionally,
4 METHOD
This work aims to generate natural and physically plausible hand-object interaction (HOI) poses, pa- rameterized by H. along with corresponding contact maps Cи and Co. conditioned on a multi-level textual prompt T and an object mesh Vo. To tackle this problem, we propose three-stage frame- work, as illustrated in Fig. 4 Specifically, the Contact Map Prediction module (Sec. 4. infers the potential contact regions on the hand and object surfaces based on the text and object geometry. The Multi-Level Conditioned Diffusion module (Sec. 4.2 synthesizes a coarse hand pose by integrat- ing coarse-to-fine textual and geometric features within diffusion framework, ensuring alignment with multi-level constraints. Finally, the Physical Constraints Refinement module (Sec. 4.3) further optimizes the coarse pose enhance contact realism and prevent penetrations.
5

---

Published as a conference paper at ICLR 2026
Input
Contact Prediction
Multi-level Conditioned Diffusion
Refinement
Raff
VAE
Last
CIM
FR
Norm
TTA
Apply [thumb pad. Index pad, ring nail] gently lift and [adge]] [plate] and then release it to drop down.
Cross Cross- Attention
Vinc
A ieer
Jayura
Adapter
Sug
Leaetner
Self-Attention
(.(x,,ty)
youh
0g
lifting up (plate]
Fall
FILM
PointNet
F
Kloc
DDPM
obj
hand
Sample Nomallza
VAE
Last
CIM
ccc
Figure 4: Overview of our three-stage framework TOUCH for generating hand-object interactions from multi-level text prompts and object meshes. CIM stands for the Condition Injection Module.
4.1 CONTACT MAP PREDICTION
To generate diverse interactions beyond simple grasping, we design two independent yet similar CVAEs (Sohn al. 2013) generate binary contact maps for the object and the hand, respectively.
For the object branch, sample point cloud Po Noxx (No 3000) from its mesh Vo. normalize it, and record the scale factor so. We use PointNet (Qi al, 2016) to extract its geometric features, which are concatenated with so form the object condition Fo. For the hand branch, generate a canonical point cloud Pf E IR Na x3 (N = 778) from MANO's ' zero pose and shape parameters H This point cloud, combined with a hand-part mask initialized from the fine-grained text DSC. is processed by PointNer to obtain the hand condition Fи- This design integrates the topological structure of the point clouds with text-guided emphasis interaction-relevant hand regions. Both CVAEs are trained conditioned on their respective geometric features (Fo, F Fx) and shared text feature F DSC (T DSC), which is extracted using the Qwen-7B (Bai et al. (203aa) processed through lightweight adapter. The optimization objective is composite loss function:
Leantact = Lifocat + Lition + LL
where L focal and Ldice supervise the contact prediction, and Lx structures the latent space. During inference, under the conditional features (Fo, FH. F FDC), the model samples from Gaussian prior ~ (0, I) and decodes it to produce the predicted binary contact maps Čo e {0, 1] 1N.1) Nox1 and C# E {0, N=x1
4.2 MULTI-LEVEL CONDITIONED DIFFUSION
The core of our method a Transformer-basee based Denoising Diffusion Probabilistic Model (DDPM) (o et al. 2020 that synthesizes hand pose parameters H conditioned on the object point cloud Po. multi-level text T, and predicted contact maps C. Instead of predicting noise, our model fa trained to directly predict the denoised data &o fo(xe,t,y) optimized with an L2 loss on the pose parameters: Lai E,, [||8 x
Condition Generation:
Transformer Inputs. To achieve precise control, our model extracts multi- level conditional features from both geometric and textual modalities. On the geometric side, we use PointNet to extract global features FO F
gib gib and point-wise local features from the object point cloud Po. the initial hand point cloud Po stage. To focus on interaction regions, we leverage Ć adaptively select features of NO = 128 object points and NH = 64 hand points near contact areas, yielding FO and FH On the textual loc loe qwen = ftezt Sscc) and fine mne-grained DSC quen text
Pt-- and the predicted contact maps Č from the previous
side, we utilize fuxt to extract both coarse- -aaseeerrined PSSC
features.
Conditional Injection: Coarse-to-Fine Control. We inject these features into the Nij blocks of our Transformer model in hierarchical, coarse-to-fine fashion. This design ensures that global

---

Published as a conference paper at ICLR 2026
context, defined by SSCs and global geometry, shapes the overall pose in early denoising stages, while local details, defined by DSCs and contact-poing features, are refined in later stages. Specifi- cally, for the i-th Transformer block:
Early Stages (i < 4): Global context is injected, with no local features.
gl concar(F gib F#f so, F (), yioe fo=00 0.
(4)
Later Stages (4 S N: Local details are injected, switching to fine- fine-grained conditions.
(5) To prevent over-reliance on any single condition and enhance robustness, we randomly drop each component of the global condition with 10% probability during training. Within each block, the global condition Yglh modulates the main features via FiLM (Perez et al. 2018), while the local
y concat( glb: so, s0, F t), yiu = concat (F loc: F kxc)
condition yioc is integrated through cross-attention to provide fine-grained spatial cues. This dual mechanism effectively decouples global contextual guidance from local geometric refinement. Fi- nally, the updated latent goes through self-attention and Feed-Forward Network (FFN).
Training Loss. To improve training stability and spatial alignment. we introduce two auxiliary losses alongside the primary diffusion loss Laitt. A global pose loss directly supervises the hand's global rotation Fix and translation T prevent overall pose drift, an issue exacerbated when directly regressing H, which comprises parameters with disparate numerical ranges (e.g., shape f3, pose O Trot. T). A distance map loss ensures precise contact by supervising the distance map dmaa E R21xNo from the 21 hand joints to the object surface. The final objective is a weighted sum:
Lotal
Laift Aglobal [foor Foc!
T + -T"
Admap ddmap
mapl
4.3 PHYSICAL CONSTRAINTS REFINEMENT
To address the common issue of global pose drift in free-form HOI generation, where the hand often fails to make contact with the object, we introduce an efficient physical refinement module. This module is powered by a refiner network, fretiner, which inherits the Transformer architecture of our diffusion model. The process begins with single forward pass to rapidly correct the global posi- tioning of the initial pose H establishing primary physical contact. Subsequently, this corrected pose undergoes Nua iterations of test-time optimization (TTA) to fine-tune local contact details, such as finger placements.
The entire optimization is guided by our self-supervised cycle-consistency loss (Caye), which en- forces bidirectional mapping consistency between hand and object contact surfaces. The core idea is that a hand contact point, after being mapped to the nearest object point via 4 (hand-to-objecd should map back to its original location via the reverse mapping V (ohject-to-hand), and vice versa. This loss acts powerful regularizer, effectively reducing the ambiguity inherent in the mappings. We combine this with Lphy (see Eq. 2 The total refinement loss is defined as:
Liefiner Lphy + Arye (EP, EPCN (P())) - Px + Ep.ePoo ((V((P))) - Polli). (7)
EXPERIMENTS
5.1 EXPERIMENTAL SETTINGS
Our experiments are conducted on the WildO2 dataset. For each hand part contact category, we perform random 4:1 split, yielding approximately 3.7k training and 677 test samples. To address the long-tailed distribution of hand part labels, we aggregate 10 less frequent hand part categories and then apply resampling using unique 7-bit labels balance the data. The model trained for 1000 epochs using the Adam optimizer with learning rate of lc-4 and batch size of 128. The diffusion model's parameters are frozen during the training of the refiner module. We evaluate our method from four perspectives: (1) Contact Accuracy, assessed by IoU and F1-score against ground-truth contacts parts. (2) Physical Plausibility, measured by Mean Per- Vertex Position Error (MPVPE). Penetration Depth (PD), and Penetration Volume (PV). Note that unlike works focusing on grasping (Jiang et al. (222I) we do not employ physics engine-base based stability simulation metrics, our scope of interactions is broader than force-closure grasps. (3) Diversity, quantified by entropy and cluster size. (4) Semantic Consistency, evaluated using a point cloud-bard based FID (P-FID) (Nichol et al. 2022 VLM assisted evaluation, and perceptual score (PS) from 10 users.
7

---

Published as a conference paper at ICLR 2026
SSCs
Obj
Ours
ContactGen Text2HOI
1hoi
Gift [rai]] completely. then it drop down.
Apply (ihumb, indes, the (nndi] rad firmly
graep
ariing and release it.
Open (tap) Apply (shumb,
and pain) rottste the
[hande]] of [tsp].
Pretend tumn [thoo] upside down, Apply middle. ring, piitky pad) to grasp the (heci] of [Shoo] and Ripping
[mnifs),
pad] evert gendl force
on the [uandl of [Unifi]
alighlly
right, Apply (indes, middls, ring, pinky pad]
fonuta) minimal
gantly
ensuring
Lift [oottin]
(thumb, Index, mid middle.
ring and piakcy pad
fimmly [Lap] of (oottle]
hold while litting one and.
Pake [bottie), Apply
(ndox
[bond]] [bottic] with quick, precise modion.
Figure 5: Visualization Results. Comparisons of different methods on samples from the WildO2 test set. Each sample consists of SSCs and an object mesh as input, with the output being an interactive hand pose. The last row shows the original authentic 2D HOI frame from internet videos.
5.2 COMPARISONS
As existing methods have not explored fine- greiggr controlled HOI generation, we select two repre- sentative types of baselines: (1) ContactGen (Liu et al., 2023b): object-conditooned multi-layer CVAE using coarse hand part labels. (2) Tex12HO1 (Cha et al. 2024): a transformer based condi- tional diffusion model guided by coarse text conditions. We remove its temporal axis and adapt it for our setting. Compared to typical grasping datasets, hand poses in WildO2 exhibit higher degrees of freedom. Both baseline methods exhibit noticeable overall hand drift. To ensure fair comparison, we also augment them with an optimization-bon based post-processing module to correct hand poses. Experimental results in Tab. show that our method outperforms baselines across most metrics. Visual results in Fig. 5 further demonstrate that our method generates more realistic HOI poses that better align with input text descriptions.
Method
Contact Ace.
Physical Plausibility
Diversity
Semantic Consistency
P-IoUt
P-FI1
MPVPE
PD]
PV1
Ent.1
CS+
P-FIDI
VLM1
PS*
ContactGen
0.620
0.730
5.46
1.296
7.37
2.85
4.93
6.08
4.8
6.3
Tex12HO1
0.711
0.795
4.69
1.239
4.93
2.85
5.20
15.72
6.5
7.5
Ours
0.776
0.844
2.97
0.932
2.67
2.93
5.40
4.13
7.1
8.8
Table 1: Quantitative comparison on HOI synthesis. We evaluate all methods on our test set us- ing comprehensive metrics covering physical plausibility. contact accuracy, diversity, and semantic consistency. 1 indicates higher is better, 4 indicates lower is better.
5.3 ABLATION STUDY
To evaluate the effectiveness our contact-guided generation spatial relations HOI and the coarse-to-fine text control design, we conduct ablation studies as shown in Tab. 2 X means without. For clarity, TTA is disabled. We argue for the primacy of contact metrics, as penetration metrics, PD and PV meaningful only after hand-object contact is established; otherwise, they be misleading. This is starkly exemplified by the "X refiner" variant, which scores poorly on contact yet achieves deceptively low PD/PV values simply because the generated hand drifts away from the
8

---

Published as a conference paper at ICLR 2026
Metrics
P-IoUt
P-FIf
MPVPE4
PD4
PVC
P-FID
Lift up [eard], applying [thumb, index pad] to geatly lift one end of the [edge] [eard] and then release drop down.
Ours(X TTA)
0.728
0.805
3.00
1.093
4.82
4.84
X hoc.
0,492
0.611
4.93
1.330
5.50
5.41
X refiner
0.513
0.621
5.05
0.723
2.98
5.84
Laye.
0.702
0.787
3.00
1.100
5.29
5.79
w/o refiner
X mul.
0.525
0.631
5.00
1.464
6.52
6.84
X TDsc
0.698
0.784
3.02
1.119
5.28
X Tssc
0.687
0.778
2.92
1.119
5.17
5.52
CLIP 0.713
0.798
2.87
1.136
4.85
4.84
w/o TTA
BERT 0.705
0.790
2.91
1.182
4.99
6.08
MPNet
0.704
0.788
2.87
1.114
5.06
6.02
Table 2: Analyzes the contributions of various components, including the absence of Mo and M, (hoc.) for guiding spatial relationship genera- tion, the multi-level network structure (mul.), and the multi-level text (x TDSC, X Tssc).
refined result
Figure 6: Contact guidance visualization.
obj mulli-view hoi
Push Igreen pen applying [hand, contact, part] exert gentle force on the [body] of Igreen pen causing slightly move.
Lift Igreen pen], applying [hand, contact, part] grip the [handle] of [green pen]
Tip Estiffed In over, applying [index, middle, ring, pinky
(a) knuckle]
firmly, ensuring a secure hold while lifting end without letting it drop.
Dopl (stuifed tuy to tip it over.
surface, (b) applying lindex, middle,, ring pad] to roll the [tip] of [nail] smoothly
Roll [mail]
the surface.
pad
thumb, Index pad
Pinch [shoel, applying [thumb, Index, middle, ring pad] opposing presoure on the Llne boc] of [shoo] to hold it firmly.
[calenlator] applying (d) lindx padl to lightly press the [ottonn] of [calculator] for precise engagement.
index pad
thumb, index, middle pad
Figure 8: Controllable Semantic Generation.
Results for green pen, demonstrating diverse and plausible hand poses generated by specify- ing different control signals, including interac- tion intent and contact geometry.
Figure 7: Out-of-Domain Generalization on
Objaverse. (a.b) Out-of-domain CAD models; (c.d) Verbs outside our primary annotated intents.
object, thus avoiding interaction entirely. This distinguishes complex task from traditional grasp- ing, where contact is facilitated by priors. The consistent degradation in contact performance upon removing any module confirms their synergistic importance. Fig, offers a qualitative visualization of the step-by-ster improvements afforded by our contact guidance.
5.4 DISCUSSION
5.4.1 ABLATION TEXT ENCODERS.
We replace the text encoder with other common token- or sentence-level encoders (e.g., CLIP (Rad- ford et al. [222], BERT (Devlin et al.) 2019), MPNet (Song et al., (020)) to analyze their impact on generation quality. Results indicate that Qwen-7B offers better performance in capturing fine- grained semantic details, detailed in Tab.
5.4.2 OUT-OR-DOMAIN GENERATION.
To evaluate generalization, we sample novel object CAD meshes from the large-scale 3D dataset Objaverse (Deitke et al. 2023). utilizing LLMs to generate captions that emulate DSC format. As visualized in Fig. / our approach successfully produces plausible interaction poses for these out-of-domain models, demonstrating strong generalization capability.

---

Published as a conference paper at ICLR 2026
5.4.3 SEMANTIC CONTROLLABILITY
Controllable Semantic Generation. Our model demonstrates strong controllability and high mantic faithfulness in interpreting fine-grained user intents. As shown in Fig. 8 by varying textual control signals (such as contact regions and interaction semanties (e.g., Push/Lift) for an object), the model successfully produces diverse and physically plausible hand poses, validating its ability to execute multi-faceted semantic directives.
Semantic Nuances of Force Expression. Although the model does not explicitly model physical forces, it learns to associate force-related terms like "firmly" and "gently" with contact geometry. generates larger, denser contacts for "firm" prompts and more marginal, sparser contacts for "gentle" ones, as shown Fig. 9 Quantitative analysis on WildO2 confirms this finding, revealing a 22-25% larger average contact area for "firm/tight" interactions.
Varying Hand Contact Parts
Varying bjjec Contact Parts
Move [box) up, applying [thumb, Indes, middle pad] to lift the [top surfaoel of box with firm grip. eusuring secure contact.
Move [fork] up, apply ing Itbumb, Index, middle padl to grasp the [bandle] of lork firmly, lifting upwards controlled
[hox] up, applying [indes, middle, ring pad] to fift the (top surface) of box gentle upw
[fork] up, applying (thumb, Inder, middle pad] to lift the handle of fork with a gentle upwar
Figure 9: Interpreting force-related semantics. The model associates "firm" prompts (Top Row) with larger contact areas and more encompassing interactions, while mapping "gentle" prompts (Bottom Row) to sparser, more marginal contact points.
CONCLUSION
In this paper, addressed the limitations of grasp-centric approaches by introducing the Free- form HOI Generation task. Our work expands the synthesis paradigm beyond simple grasping to broader, semantically expressive spectrum of interactions. To support this, built an automated pipeline to construct WildO2, an in-the-wild 3D dataset for daily HOIs, providing a critical resource enable future research in this domain.
Limitations and Future Directions. Our framework currently focuses on static HOI snapshots, which inherently limits its ability capture the temporal dynamics of an interaction process. While our pipeline offers rapid expansion, the current dataset scale also presents an area for future growth. In the future, we plan to extend our work dynamic sequences by leveraging large-scale video datasets and incorporating 6- D-DoF object pose estimation, thus modeling the entire human- environment interaction process.
7 ACKNOWLEDGMENTS
This work supported by the National Natural Science Foundation of China (NSFC) under Grants 62225207, 62436008, 62306295, 62576328, and 625B2175. The Al-driven experiments, simula- tions and model training were performed on the robotic Al-Scientist platform of Chinese Academy of Sciences.
REFERENCES
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chenggiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang. Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang
10

---

Published as a conference paper at ICLR 2026
Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. arXiv preprint arXiv:2309. 2023a.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision- languae model for understanding, local- ization, text reading, and beyond. arXiv preprint arXiv:2308. /2966, 2023b.
Samarth Brahmbhatt, Chengcheng Tang, Christopher D Twigg, Charles C Kemp, and James Hays. Contactpose: A dataset of grasps with object contact and hand pose. In European Conference on Computer Vision, pp. 361-378 Springer, 2020.
Zhe Cao, Ilija Radosavovic, Angjoo Kanazawa, and Jitendra Malik. Reconstructing hand-object interactions in the wild. ICCV, 2021.
Junuk Cha, Jihyeon Kim, Jae Shin Yoon, and Seungryul Back. Text2hoi: Text-guided 3d motion generation for hand-object interaction. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1577-1585, 2024.
Hongyi Chen, Yunchao Yao. Yufei Ye, Zhixuan Xu, Homanga Bharadhwaj. Jiashun Wang, Shubham Tulsiani, Zackory Erickson, and Jeffrey Ichnowski. Web2grasp: Learning functional grasps from web images of hand-object interactions, 2025. URL htttps://arxii> org/ obg.///////.......
05517
Sammy Christen, Shreyas Hampali, Fadime Sener, Edoardo Remelli, Tomas Hodan, Eric Sauser, Shugao Ma, and Bugra Tekin. Diffh2o: Diffusion- based synthesis of hand-object interactions from textual descriptions. In SIGGRAPHI Asia 2024 Conference Papers, pp. 1-11, 2024.
Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26, 2013.
Dima Damen, Hazel Doughty, Giovanni Maria Farinella. Sanja Fidler, Antonino Furari,, Evange- los Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. The epic-kitchens dataset: Collection, challenges and baselines. IEEE Transactions Pattern Analysis and Machine Intelligence (TPAMI), 2020.
Matt Deitke, Ruoshi Liu, Matthew Wallingford, Huong Ngo, Oscar Michel, Aditya Kusupati, Alan Fan, Christian Laforte, Vikram Voleti, Samir Yitzhak Gadre, Eli VanderBilt, Aniruddha Kembhavi, Carl Vondrick, Georgia Gkioxari, Kiana Ehsani, Ludwig Schmidt, and Ali Farhadi. Objaverse-xl: A universe of 10m+ 3d objects. arXiv preprint arXiv:2307.0560 2023.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume (long and short papers), pp. 4171-4186, 2019.
Johan Edstedt, Qiyu Sun, Georg Bökman, Marten Wadenbäck, and Michael Felsberg, RoMa: Robust Dense Feature Matching. IEEE Conference on Computer Vision and Pattern Recognition, 2024.
David Eigen, Christian Puhrsch, and Rob Fergus. Depth map prediction from single image using multi-scall deep network. Advances in neural information processing systems, 27, 2014.
Zicong Fan, Maria Parelli, Maria Eleni Kadoglou, Xu Chen, Muhammed Kocabas, Michael J Black, and Otmar Hilliges. Hold: aategory-agnoste 3d reconstruction of interacting hands and objects from video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 494-504, 2024.
Hao-Shu Fang, Chenxi Wang, Minghao Gou, and Cewu Lu. Graspnet- 1biliionn A large-scale bench- mark for general object grasping. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11444-111 11453, 2020.
Thomas Feix, Javier Romero, Heinz-Bodo Schmiedmayer, Aaron M Dollar, and Danica Kragic. systems, 46
The grasp taxonomy of human grasp types. IEEE Transactions on (1):66-77,, 2015.
11

---

Published as a conference paper at ICLR 2026
Rao Fu, Dingxi Zhang, Alex Jiang, Wanjia Fu, Austin Funk, Daniel Ritchie, and Srinath Sridhar. Gigahands: massive annotated dataset of bimanual hand activities. In Proceedings of the Com- puter Vision and Pattern Recognition Conference, pp. 17461-17474, 2025.
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski. Joanna Materzynska, Susanne West- phal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. The "something something" video database for learning and evaluating visual common sense. In Proceedings of the IEEE international conference on computer vision, pp. 5842-5850, 2017.
Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Gird- har, Jackson Hamburger, Hao Jiang. Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition, pp. 18995-19012, 2022.
Shreyas Hampali, Mahdi Rad, Markus Oberweger, and Vincent Lepetit. Honnotate: A method for 3d annotation of hand and object poses. In CVPR, 2020.
Yana Hasson, Güil Varol, Dimitris Tzionas, Igor Kalevatykh, Michael J. Black, Ivan Laptev, and Cordelia Schmid. Learning joint reconstruction of hands and manipulated objects. In CVPR, 2019.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-68511 2020.
Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. Lrm: Large reconstruction model for single image to 3d, 2024. URL https:// /awpvvaaaivvvv//a org/ abs 2201/ 04400
Hanwen Jiang, Shaowei Liu, Jiashun Wang, and Xiaolong Wang. Hand-object contact consistency reasoning for human grasps generation. In Proceedings of the International Conference on Com- puter Vision, 2021.
Korrawe Karunratanakul, Jinlong Yang. Yan Zhang, Michael Black, Krikamol Muandet, and Siyu
Tang. Grasping field: Learning implicit representations for human grasps, 2020. URL https: / //aaxiv.org/aabbaba abs///////a//// wtt.bb//2000.. 04451
Hongxiang Li. Yaowei Li, Yuhang Yang. Junjie Cao, Zhihong Zhu, Xuxin Cheng, and Long Chen. Dispose: Disentangling pose guidance for controllable human image animation. arXiv preprint 244.:24/2,009449, 2024a.
Kailin Li, Jingho Wang, Lixin Yang, Cewu Lu, and Bo Dai. Semgrasp: Semantic grasp generation via language aligned discretization. In European Conference on Computer Vision, pp- 109-127. Springer, 2024b.
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-l-to-3: Zero-shot one image to 3d object. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9298-9309, 2023a.
Shaowei Liu, Yang Zhou, Jimei Yang. Saurabh Gupta, and Shenlong Wang. Contactgen: Generative contact modeling for grasp generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 20609-20620, 2023b.
Yumeng Liu, Xiaoxiao Long, Zemin Yang, Yuan Liu, Mare Habermann, Christian Theobalt, Yuexin Ma, and Wenping Wang. Easyhoi: Unleashing the power of large models for reconstructing hand-object interactions in the wild. arXiv preprint arXiv:2411.14288 142280, 2024a.
Yunze Liu, Yun Liu, Che Jiang, Kangbo Lyu, Weikang Wan. Hao Shen, Boqiang Liang, Zhoujie Fu, He Wang, and Li Yi. Hoi4d: A 4d egocentric dataset for category- levell human-object interac- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 21012-21022, June 2022.
Zhiheng Liu, Ka Leong Cheng, Qiuyu Wang, Shuzhe Wang, Hao Ouyang, Bin Tan, Kai Zhu, Yu- jun Shen, Qifeng Chen, and Ping Luo. Depthlab: From partial to complete. arXiv preprint arXiv:2412.18/15 18/53, 2024b.
12

---

Published as a conference paper at ICLR 2026
Zhiheng Liu, Hao Ouyang, Qiuyu Wang. Ka Leong Cheng, Jie Xiao, Kai Zhu, Nan Xue, Yu Liu, Yujun Shen, and Yang Cao. Infusion: Inpainting 3d gaussians via learning depth completion from diffusion prior. arXiv preprint arXiv:2444.1611, 2024c.
Joanna Materzynska, Tete Xiao, Roei Herzig, Huijuan Xu, Xiaolong Wang, and Trevor Darrell. Something-else: Compositional action recognition with spatial-temporal interaction networks. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 1049- 1059, 2020.
Alex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen. Point-e: A system for generating 3d point clouds from complex prompts, 2022. URL
/rg/
abs 2ttp/2212.000 08751
Austin Patel, Andrew Wang, Ilija Radosavovic, and Jitendra Malik. Learning to imitate object interactions from internet videos, 2022. URL/attss://axx// wwg.or///aaoo. abs /222221111... 13225
Georgios Pavlakos, Dandan Shan, Ilija Radosavovic, Angjoo Kanazawa, David Fouhey, and Jitendra
Malik. Reconstructing hands in 3D with transformers. In CVPR, 2024.
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018
Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv, 2022.
Charles R Qi Hao Su, Kaichun Mo, and Leonidas Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. arXiv preprint arXiv:1612.00593 2016.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. CoRR, abs/2103.00020,, 2021. URL https:::: httttps:/aaxxiv.ooo org /tct/ 0000/0000.00000000 0002C
Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Va- sudev Alwala, Nicolas Carion, Chao-Yuu Yuan Wu, Ross Girshick, Piotr Dollár, and Christoph Fe- ichtenhofer. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714 2024. btLpstps: www.aaavvvv... org abs / 408 00714
Javier Romero, Dimitrios Tzionas, and Michael J. Black. Embodied hands: Modeling and capturing hands and bodies together. ACM Transactions on Graphics, (Proc. SIGGRAPH Asia), November 2017. PtR:ttt:://dl acm. 10. 11....11151111330 wttpa..gg/1...111/33333333333333
3130883
Dandan Shan, Jiaqi Geng, Michelle Shu, and David Fouhey. Understanding human hands in contact at internet scale. 2020.
Kihyuk Sohn, Honglak Lee, and Xinehen Yan. Learning structured output representation using deep conditional generative models. Advances in neural information processing systems, 28, 2015.
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie Yan Liu. Mpnet: Masked and permuted pre- training for language understanding. arXiv preprint arXiv:2004.09297, 2020.
Omid Taheri, Nima Ghorbani, Michael Black, and Dimitrios Tzionas. Grab: A dataset of whole- body human grasping of objects. European conference on computer vision, pp. 581-600. Springer, 2020.
Chao Tang, Dehao Huang, Wenqi Ge. Weiyu Liu, and Hong Zhang. Graspgpt: Leveraging se- mantic knowledge from large language model for task-oriented grasping. IEEE Robotics and Automation Letters, 8(1 8(11):5511-5588 2023.
Tencent Hu Team. Hunyuan3d 2.5: Towards high-fidellnn fidelity 3d assets generation with ulti- mate details, 2025. URL //axxivv wwtp://wwwww/////////e// 16004]
13

---

Published as a conference paper at ICLR 2026
Yongqi Tian, Xueyu Sun, Haoyuan He, Linji Hao, Ning Ding, and Caigui Jiang. Funhoi: Annotation-free 3d hand-object interaction generation via functional text guidane, 2025. URL
https://arxiv: org/ abs / 2502. 2502.20805
Yi-Lin Wei, Jian-Jian Jiang. Chengyi Xing, Xian-Tuo Tan, Xiao-Ming Wu, Hao Li, Mark Cutkosky, and Wei-Shi Zheng. Grasp as you say: Language-guidee guiddd dexterous grasp generation. Advances in Neural Information Processing Systems, 37:46881-46907, 2024.
Boran Wen, Dingbang Huang, Zichen Zhang, Jiahong Zhou, Jianbin Deng, Jingyu Gong, Yulong
Chen, Lizhuang Ma, and Yong-Lu Li. Reconstructing in-the-wild open-vocabulary human-object interactions, 2025. URL URRittts::/axxii.oooo
abc///bs//220 220../5533.20 15898
Jane Wu, Georgios Pavlakos, Georgia Gkioxari, and Jitendra Malik. Reconstructing hand-held objects in 3d. arXiv preprint wwxiv:240.006507 2024.
Jiale Xu, Weihao Cheng, Yiming Gao, Xintao Wang. Shenghua Gao, and Ying Shan. Instantmesh: Efficient 3d mesh generation from a single image with sparse-view large reconstruction models. arXiv preprint arXiv:2404.07199, 2024.
Lixin Yang. Xinyu Zhan, Kailin Li, Wenqiang Xu, Jiefeng Li, and Cewu Lu. CPF: Learning contact potential field to model the hand-object interaction. In ICCV, 2021.
Lixin Yang, Kailin Li, Xinyu Zhan, Fei Wu, Anran Xu, Liu Liu, and Cewu Lu. Oakink: A large-scale interaction. In Proceedings of the IEEE/CVF
knowledge repository for understanding conference on computer vision and pattern recognition, pp. 20953-20962, 2022.
Yuhang Yang, Wei Zhai, Hongchen Luo, Yang Cao, Jiebo Luo, and Zheng-Jun Zha. Grounding 3d object affordance from 2d interactions in images. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 10905-10915, October 2023.
Yuhang Yang, Wei Zhai, Hongchen Luo, Yang Cao, and Zheng-Jun Zha. Lemon: Learning 3d human-object interaction relation from 2d images. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. (6284-16295, 2024a.
Yuhang Yang, Wei Zhai, Chengfeng Wang, Chengjun Yu, Yang Cao, and Zheng-Jun Zha. Ego- choir: Capturing 3d human-object interaction regions from egocentric views. arXiv preprint arXiv:2405.13659, 2024b.
Yufei Ye, Abhinav Gupta, and Shubham Tulsiani. What's in your hands? 3d reconstruction of generic objects hands. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3895-3905, 2022.
Yufei Ye, Abhinay Gupta, Kris Kitani, and Shubham Tulsiani. G-hop: Generative hand-object prior
for interaction reconstruction and grasp synthesis. In CVPR, 2024.
Chengjun Yu, Wei Zhai, Yuhang Yang, Yang Cao, and Zheng-Jun Zha. Hero: Human reaction
generation from videos. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10262-10274, 2025.
Xinyu Zhan, Lixin Yang, Yifei Zhao, Kangrui Mao, Hanlin Xu, Zenan Lin, Kailin Li, and Cewu Lu. Oakink2: dataset of bimanual hands-object manipulation in complex task completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 445 456, 2024.
Zhenhao Zhang, Ye Shi, Lingxiao Yang, Suting Ni, Qi Ye, and Jingya Wang. Openhoi: Open- world hand-object interaction synthesis with multimodal large language model. arXiv preprint arXiv:2505.18% 18944, 2025a.
Zhongqun Zhang, Hengfei Wang. Ziwei Yu, Yihua Cheng, Angela Yao, and Hyung Jin Chang. N12contact: Natural language guided 3d hand-object contact modeling with diffusion model. In European Conference on Computer Vision, pp. 284-300. Springer, 2024.
Zichen Zhang, Hongchen Luo, Wei Zhai, et al. Pear: phrase- phrase-based cand-object interaction anticipa- tion. Science China Information Sciences, 68( 68(150209) 2025b. doi: 10. 1007/s/ 1eess114449 032-0044444
14

---

Published as a conference paper at ICLR 2026
Ying Zheng, Lei Yao. Yuejiao Su, Yi Zhang, Yi Wang, Sicheng Zhao, Yiyi Zhang, and Lap-Pui Pui Chau. A survey of embodied learning for object-centric robotic manipulation. Machine Intelligence Research, pp. 1-39, 2025.
A APPENDIX
A.1 MORE EXPERIMENT RESULTS
A.1.1 MORE ABLATION EXPERIMENTS
Action-Specific Analysis We performed a per-category statistical analysis of the metrics for the most frequent verb categories in the set, detailed Tab.
3 The results reveal significant
variations across different verb categories, reflecting the distinct characteristics of each action.
Push
Poke
Lift
Pick
Move Turn
Pull
Put
Open
Roll Spin
P-IoU
0.762 0.811 0.805 0.752 0.773 0.725 0.830 0.753 0.778 0.739 0.763
P-FI
0.833
0.852 0.872 0.834 0.844 0.821 0.891 0.828 0.866 0.819 0.834
MPVPE
3.098 3.458 2.724 2.511 2.618 3.014 2.301 3.034 2.621 2.653 3.844
PD
1.135
0.391 0.568 1.127 0.811 1.462 1.214 1.105 1.206 1.294 0.892
PV
3.025 0.922 1.716 2.814 2.842 5.718 3.440 3.656 4.081 1.652 3.052
Table 3: Per-category performance metrics for the most frequent verb categories in the test set.
For instance, the Poke category, which typically involves singular or highly localized contact area, consequently yields higher scores on contact-related metrics such as P-IoU and P-FI. However, this action imposes fewer constraints on the overall hand orientation and the configuration of non- contacting fingers, leading to a higher MPVPE for the generated hand poses. Conversely, the Lift category, which closely resembles a simple grasping motion, is generally less complex. As result, demonstrates strong performance across most metrics, which corroborates our earlier assertion regarding the relative simplicity of grasping operations. In general, for given verb, greater diversity in the associated actions correlates with increased learning difficulty for the model.
Layer Injection Strategy We conducted an ablation study the coarse-to-fine layer injection strategy (Tab. 4p to justify the chosen 4/4 split against alternatives (e.g., 0/8, 2/6, 6/2).
Split Config
P-IOU 1
P-F1 1
MPVPE
PD J
PV 1
0/8 (x mul)
0.766
0.836
2.97
0.950
2.55
2/6
0.767
0.839
2.97
0.942
2.52
4/4 (Ours)
0.776
0.844
2.89
0.932
2.67
6/2
0.650
0.747
3.64
1.026
4.46
Table Ablation on Coarse/Fine Layer Split. Comparison different layer allocation strategies for the coarse-to-fine injection. The 4/4 split yields the best trade-off between semantic structure and geometric precision.
The results demonstrate two key points: (1) Optimal Balance: The 4/4 configuration achieves the best overall performance, confirming that a balanced injection strategy effectively handles both global semantics and local contact details. (2) Fine-grained Priority: The 2/6 split (late fine- fineggrained injection bias) significantly outperforms the 6/2 split (early coarse injection bias). This indicates that for generating free-form HOI, conditioning fine-grained information is more critical than early global coarse cues.
Impact of Detailed Contact Directives. To qualitatively illustrate the findings from our ablation study (Tab. 2). we visualize the model's behavior when fine-grained contact directives are omit- ted. As shown in Fig. 11] without this guidance, the model tends to collapse to common grasping
15

---

Published as a conference paper at ICLR 2026
WildO2
TOUCH
WildO2
TOUCH
spin pen and sloa.spplving Index nalls middle knuckla, ding apply sudden force to halt the motion.
Tipping box averyappyying index knuckle, middle knuckle, (ind
over,
More Contacts
Brabing lemon qugueeee so that almost falls aff but exert contralied force on the handle lemon squeecer, guidling
Push wallet-ppryyngg lndic nell wéddle ped excrt facce on the edge nalet
Pusk extectial glss leen an to right apptring make sall eddile keacclle one Interal
buttin.sphhing
spray alghtly nows
right midds piekoy bettery, galding smoettly pentis the surface.
cerreteu Mllanshing thunb knaoies Indes ped ped, ring ped perity
peet, hin,applying thumb Index
More Objects
waste bin fimely, ensuring secure held while lifting
gramp
üit business undaareling thand pas, indes pad to gastly grsp the edge esloo
scals emutig &t abseat Usa bet remeisa belanced.
cyliedrical can with minimmll force.
Lifting up toy cor-applying thumb padi, index pad to lift one end
LR ous-golsing Dhae sad lndox pari rekidie pad, pod to securely grop the
More Actions
Push autbett.aabbig
pentie
faditats movereest.
Twist loc aet.applyyin pad. pad to rotabe the kcob with pra-
doorknob ped, palm grip rotate Index plincy controlled force.
apotsing pad ped, reddlle pad, platy
bottle mederate fecce
Balling lighipter.apptiig indos pad, rvidile ring tortall the body
Poli bill.auelying thumis paid, Index pail to carefully align and
Figure 10: Visualization Results. More samples from the WildO2 test set. For each sample, the leftmost image represents the hand-object interaction frame (Thoi) extracted from Internet videos. The middle image shows our reconstruction of Thois which serves as the ground truth for WildO2. The rightmost image illustrates the free-form HOI generated by our method, TOUCH, conditioned on object meshes and DSCs. The bottom row contains the corresponding DSCs.
patterns, even for non-grasping prompts like "put tea bag surface". This underscores the neces- sity of detailed contact information for generating nuanced and accurate interactions beyond simple grasps.
16

---

Published as a conference paper at ICLR 2026
GT
DSC
SSC
Put Itea bagl surface. Applying [thumb pad, middle naill to gently place the [body] of bag on flat surface.
Put [tea bag] surface.
Figure 11: Comparison of generations with and without detailed contact guidance. (Left) Given a Short Synthetic Caption (SSC) that omits contact information, the model defaults to a generie and incorrect grasping pose. (Right) With Detailed Synthetic Caption (DSC) specifying the intended contact points, the model successfully generates the correct, nuanced "placing" interaction.
A. 1.2 MORE COMPARISON EXPERIMENTS
We evaluate our method on the OakInk dataset, as detailed in Tab Using its CapGrasp (Li et al. 2024b) annotations and computed part contact information, we construct fine- linegrained text conditions through summarization via Qwen (Bai et al. 2023a and verification.
Method
Contact Ace.
Physical Plausibility
Diversity
Semantic
P-IoUt
P-FI
MPVPEL
PDL
PVL
Ent.1
CSt
P-FID
ContactGen
0.654
0.769
6.69
0.790
16.59
2.93
5.33
11.25
Tex12HOI
0.778
0.860
5.49
0.807
7.05
2.84
3.48
4.95
Ours
0.812
0.882
4.49
0.939
5.89
2.92
3.50
3.21
Table 5:
Quantitative comparison on hand-object interaction synthesis in OakInk-Shape dataset. f indicates higher is better, 4 indicates lower is better.
A.1.3 FAILURE CASES ANALYSIS
Despite its overall strong performance, our analysis reveals several recurring failure modes in the TOUCH model. These can be categorized into four main types, as illustrated in Fig. [2
Pose Bias: The model often defaults to a grasping pose, even when textual cues contradict this action (e.g., prompting gentle touch without thumb).
Orientation Error: The generated hand exhibits an incorrect or unnatural palm orientation relative to the object.
The guided contact between the hand and the object is misplaced, leading to physically implausible interactions.
Contact Mismatch:
Penetration Artifacts:
The model produces significant penetration of the fingertips into the object's surface.
A.2 DETAILS OF WILDO2 ACQUISITION
A.2.1 SCALABILITY OF THE PIPELINE
Our pipeline's design emphasizes three key properties: evolvability, allowing it improve with stronger upstream models; scalability, enabling rapid data reconstruction from web-scale clips ap- proximately less than 2 minutes per case; and controllability, ensured by our rigorous human-in- the-loop quality control.
Evolvability with Upstream Image-to-3D Models.. Our pipeline's performance directly cou- pled with the upstream image-to-3D model, modular design that allows us to benefit from ongoing advancements. We demonstrate this by replacing our default backbone, InstantMesh, with the more powerful Hunyuan3D 3.0 (Team 2025) As shown in Fig, [3] this upgrade yields significant improve- ments. For samples that were already successfully reconstructed (first two rows), the new backbone enhances geometric details, such refining object edges and correctly modeling the internal cavity
17

---

Published as a conference paper at ICLR 2026
DSCs
hoi
Wild02
TOUCH
Push [bttle)), applying [lndex, middle
(a) and ring pad] exert gentle force [body] [bottlel, causing It to slightly without Falling.
Push Icylindrical bottle), applying
[thumb, index, middle, ring and pinky
(b) pad] exert geatle foroe the [body] of (sylindrical bottle), guiding towards
the odge without causing it to fall.
(c) Pake [bos] applying [index pad] to gently prod the (side] [box], cansing to slightly shift.
Put fmushroom lamp] on surface, applying [thumb, indes, middle, ring pad and palm] to support the [base] of [mushroom lamp] place steadily on the surface.
Figure 12: Qualitative examples typical failure modes of the TOUCH model. Each panel corresponds to category discussed in the text: (a) bias towards grasping poses, (b) incorrect palm orientation, (c) misplaced contact guidance, and (d) significant fingertip penetration.
of a cup. More importantly, it successfully reconstructs instances that had previously failed (third and fourth rows), thereby increasing the overall success rate of our pipeline. Nevertheless, chal- lenges persist even for advanced models, such as reconstructing transparent objects or fine-grained structures, highlighting the continued potential for future advancements to further enhance the qual- ity of WildO2.
lee
Jnni
InstanMMesh
Wild02
Humpann
wadoz'
Succesful sumpale
Folliuro sample
Figure 13: Qualitative comparison with stronger reconstruction backbone. WildO2* denotes results using the Hunyuan3D 3.0 backbone, The first two rows show refinements previously suc- cessful reconstructions, (e.g., sharper details, correct topology) while the bottom two rows demon- strate recovery of previously failed cases. This highlights the pipeline's ability to evolve with ad- vancements image-to-3D technology.
Computational and Time Cost Analysis. On one NVIDIA A40 GPU, our automated steps are efficient: 3D object reconstruction averages 17s per sample with InstantMesh, (ranging from seconds to 3 minutes for other backbones) and our alignment module takes 14s. The subsequent manual verification and refinement, applied to every sample, requires an average of 50 seconds (std = 49s) per person. This commitment to manual oversight is crucial investment in the final dataset's quality and physical plausibility.
Human-in-the-Loog Quality Control. To ensure the highest data fidelity, every sample undergoes mandatory human review. This process guided by automated quality metrics (CLIP similarity, 2D rendering scores) that assist annotators. Each review results in of four outcomes:
18

---

Published as a conference paper at ICLR 2026
Accept: The reconstruction meets all quality standards.
Refine: Minor inaccuracies are corrected using our dedicated UI (Fig. 14 left).
Re-generate: A low-quality sample is re-processed with an alternative image-to-3D model.
Discard: Intractable cases that fail to meet quality standards are removed.
A.2.2 RECONSTRUCTION ANALYSIS
The primary obstacle is Geometric Reconstruction Failure, where 3D mesh recovery unsuccessful. Another category, Non-Interactiv Cases, is defined for instances where the 2D HOI guide image itself showed no discernible contact. Explicit failures in Pose Estimation account for 2.1%. Finally, the Other Cases category contains various failures that bypassed initial rule-based screening: these include some of the most complex scenarios, such as interactions involving deformable or transpar- ent objects, which fall outside the scope of our current reconstruction method. Details and examples of each failure type are provided in Fig. 14
Figure 14: Examples of reconstruction failures. (Left) A screenshot of our annotation UI show- ing "Non-Interactive" failure, where the reconstructed hand and object do not interact. (Right Top) Examples of "Geometric Reconstruction Failure" for the object and the hand. This was the most common failure category, with object geometry being more challenging to reconstruct than the hand's. (Right Middle) Examples of "Pose Estimation Failure", where the 6DoF pose of the hand or object was incorrectly recovered. (Right Bottom) Examples of "Other Cases". The first exam- ple shows a deformable object, and the second shows an object that is part of larger structure (a drawer).
A.2.3 SELECTION OF RECONSTRUCTABLE CLIPS FROM LARGE-SCALE IN-THE-WIH WILD VIDEOS
Our data reconstruction and annotation process commences with the SonnethiggSSometiigg V2 (SSv2) dataset (Goyal et al. 2017). a large-scale collection of over 220,847 videos. We chose this dataset for several key reasons. First, its vast scale and diversity, resulting from a crowdsourced effort where contributors enact specific verb-noun prompts, provide a rich variety of goal-oriented hand-object interactions. Second, many videos feature close-up shots of the hand-held object, which is advantageous for 3D reconstruction. Critically, the associated Something- ska al.) 2020) furnishes bounding box annotations for hands and objects approximately 180,049 of these clips, offering invaluable starting point. However, primary challenge of SSv2 is its low resolution (typically 240 pixels in height), which necessitates a rigorous filtering pipeline to isolate clips suitable for high-fidelity reconstruction.
dataset (Materzyn-
To ensure the viability of our reconstruction pipeline, we implemented multi-stage filtering process to distill the initial 180k annotated clips into high-quality subset. The screening criteria were de- signed to simplify the interaction scenario (e.g., single hand, single object) and guarantee sufficient visual quality for reliable 3D modeling. This process effectively removes clips that ambigu- ous, occluded, or otherwise intractable. The detailed steps and their impact on the dataset size are summarized in Tab. 6
19

---

Published as a conference paper at ICLR 2026
Table 6: The multi-stage filtering pipeline for selecting reconstructable clips from the Something- Something V2 dataset. The process starts with clips annotated by Something-EAA -lse and progressively refines the selection based on interaction complexity and visual quality.
Step
Criterion
Description
Clips Remaining
1
Initial Annolated Set
Clips from SSv2 with hand and object bounding box anmotations. Exclude clips involving multi hands to simplify interaction modeling.
180,049
2
Single-Hand Interaction
137,578
3
Single-Object Interaction
Retain only clips where single object is being munipulated.
82,728
4
Minimum Object Size
Discard clips where object too small to be reliably reconstructed.
57,384
5
Object Visibility
Exclude clips where the object is too large or moves out of frame.
12,888
6
Trame Slability
Ensure stable pre-inleracttin and interaction frumes be identified.
8,551
A.2.4 ACQUISITION OF O2HOI FRAME PAIRS
This section details the automated procedure for extracting an "Object-only to Hand-Object Interac- tion" (O2HOI) frame pair, denoted as (Iet /no)), from each video clip.
1. Mask Generation and Frame Selection Preliminaries. For each frame in a given video sequence, we generate object masks (Mo) and hand masks (MHr) using the SAM2. The interaction period, TTHo = [anaa ) is defined as the contiguous block of frames where the Intersection over Union (IoU) between the object mask and the hand mask, each expanded by several pixels, exceeds predefined threshold. The reference frame, Leff is selected as the object-only frame temporally closest to this interaction period.
2. Optimal HOI Frame Selection. The primary objective is to select an index thoi from ThOI such that the object's pose in frame Ioo has undergone minimal change relative to its pose in Let- We achieve this by estimating the 2D affine transformation between the object in the reference frame and in every candidate frame within THoi- This estimation utilizes the RoMa dense feature matcher (Edstedt et al., (2024). The detailed steps for comparing a candidate frame I (where f E THOi) to the reference frame Lef
follows:
Feature Matching: For each pair (Teef It), we extract robust keypoint correspondences (Kref, k,) within their respective object masks, Me and MP. Transformation Estimation: The keypoints are used to robustly estimate an affine transfor- mation matrix A, using RANSAC.
Rotation and IoU Calculation: The rotation angle sequence 6, and IoU metrics com- puted as follows:
Rs A,[0: 2.0:2].
U.S.V = SVDD(ss), R-UVT,
(8)
180 IoU,- |warpAffine (M M A,) n
0. arctan 2(R21 R1))
(9)
π
|warpAffine A.) U M,
Interaction Frame Selection: with IoU gradient AloU, The selected HOI frame index Ahoi is determined by:
arg mineeTRe |0el. arg min It imaxl;
if minke |0.| MAX MIN ANGLE
if max [6 MIN MMA.ANNLE
arg min
TETHO
It imax
otherwise
10.1 MMX. ANGLE
(10)
{t THOI AloU, < IHHRUS and ANGLEE, MIN MAX ANGLE=5
where Satable
Inpainting Mask Generation: In summary, obtain the 02HOI frame pair (fret, thoi) and the corresponding inpainting mask for subsequent processing:
Mmpaint warpA PrrpAAffineMM.. Aboi).
(11)
This procedure ensures accurate spatial alignment between the object-only and interaction frames, facilitating efficient and reliable inpainting mask generation for downstream tasks.
20

---

Published as a conference paper at ICLR 2026
A.2.5 ESTIMATION OF OBJECT ELEVATIONS
The backbone of Lu al. 2024), Zero123 al. 2023a). generates novel views of objects; however, the predicted elevation angles may not precisely align with the control signals. To address this, we employ rendering based matching strategy estimate the optimal elevation angle for the reconstructed 3D object mesh. Specifically, we render the mesh from candidate elevations and select the angle whose 2D projection best matches the HOI frame, providing reliable initial- ization for subsequent optimization. The search is performed in two stages: (a) coarse search over the range [-40%, 60'] with sampled angles, and (b) a fine search within [best 10", best + 10"] using 21 samples. The selection criterion is the point-wise mean squared error (MSE) between the rendered projection and the aligned HOI frame.
N Σ (Tarl Trendered (0)
elevation = arg min lEe N
(12)
We employ the CL.PP-similarity metric automatically assess the reconstruction quality of image- to-3D methods. This enables efficient and coarse filtering of clips, ensuring that only those with high-quality and reliable 3D reconstructions are selected for subsequent processing and annotation.
A.2.6 CONTACT COMPUTATION
Given an object point cloud E IR N.3 and hand point cloud H R No x3, we propose a robust four-stage contact map computation algorithm:
Stage 1: Bidirectional Nearest Neighbor Voting. For each point, we compute the nearest neighbor in the opposite set and accumulate votes:
vote, = Σ INN(())) = =.), vote: Σ I[NN(o:) =hy]-
Stage 2: Candidate Selection. High-frequeng candidate points are selected based on the upper quantile of the vote distribution:
Co = (i) 2 Q1-a(vote.))
Stage 3:
Distance Validation. Core contact points are further filtered by requiring their hand-object distance de to be below both quantile threshold 3 and absolute threshold
S, {i E C. d; < min (2(d),
Stage 4: Region Expansion. The contact region is expanded by including points within a radius 7 d of any core contact point, where d is the mean k-nearest neighbor distance:
M. = {i min |o; -0 S SES,
A.2.7 WILDO2 DSCS PROMPT
(structured Ousput Protecol] As a hand-object interaction analyzer, generate JSCN atrictly like this:
"obi "obi_categorr"; "eylindrical", "general"; "grip lobi_categoryl", "physical"; "Apply [hand_sontact] to astablish stable three-point contact with [obi_contact] of [obj_categozy. while other fingers form loose sphere.", "hand_contact"" ["thumb pad", "pinky nail", "palm"), "obj_contac:"; "body"
[Key Requirements:
1. Mandatory placeholders: [hand_contact]] [obj_contact), [obi_category]
Pay Field Rules:
21

---

Published as a conference paper at ICLR 2026
[aand_contact,, placeholder ABSOLUTELY explicit contact point erumeration
must describe contact areas, interaction methods, and force levels considering object functionality may
3. hand_contact is where interaction is most likely to occur. Input hand_eentact. sensing errors from point cloud enalysis. Valid hand_contaact options (17 total): (' thumb p'd","indas" 'index pad', 'miidde pad", rring pad", 'picky pad', , ttumb nail', , iidex nail', , 'middle nail', "oll,rra ring nail', /pinky nail', , thumb knuckle', , index knuckle', /'iidie knuckle', ring knuckle', 'pinky knuckle', 'palm', /back palm',
ARSOLUTELY PRGAMR1E/200NN0TES, only
output
Current Input Interaction Description: "hand_contast"; xxx, XXx.
A.2.8 DATASET ANALYSIS
For data integration purposes, Tab. outlines the mapping used to align labels from the Something- Something dataset with our defined action groups. A comprehensive comparison with existing hand- object interaction datasets is presented in Tab, 8 We further analyze the statistical properties of our collected objects, including the scale distribution and its relationship with action categories, as visualized in Fig. 15
Table 7:
The correspondence between the class labels of the Something -oatiiinnn- dataset and the action groups defined in this study.
Action Group
Class Label
Pretending to pick something up
Picking
Picking something up
Pushing something from right to left
Pushing something from left to right
Pushing something so that it almost falls off but doesn't
Pushing something that it slightly moves
Pushing something spins
Pushing
Pushing something so that it falls
off the table
Pushing something off of something
Pushing something onto something
Pushing something with something
Something colliding with something and both are being deflected
Poking something so that it falls over
Poking stack of something the stack collapses
Poking something so slightly moves
Poking something so lightly that it doesn't or almost doesn't move
Poking
Pretending to poke something
Poking something so that it spins around
Poking a stack of something without the stack collapsing
Poking a hole into something soft
Poking a hole into some substance
Lifting up end of something without letting drop down
Lifting something up completely without letting it drop down
Lifting up one end of something, then letting it drop down
Lifting
Lifting something up completely, then letting drop down
Lifting something with something on it
22

---

Published as a conference paper at ICLR 2026
Lifting surface with something on it but not enough for it to slide down
Moving something down
Moving something towards the camera
Moving something away from the camera
Moving
Moving something across surface without it falling down
Moving something up
Moving something across a surface until falls down
Moving part of something
Pretending squeeze something
Squeezing
Squeezing something
Taking one of many similar things on the table
Pretending take something from somewhere Pretending take something out of something Taking something from somewhere
Taking
Touching
Touching (without moving) part of something
Opening something
Opening
Pretending to open something without actually opening
Laying
Laying something on the table on its side, not upright
Pretending turn something upside down
Turning
Turning something upside down
Rolling something flat surface
Rolling
Letting something roll along flat surface
Letting something roll down a slanted surface
Spinning something that quickly stops spinning
Spinning
Spinning something it continues spinning
Pulling something from right to left
Pulling
Pulling something from left to right
Throwing something
Throwing
Throwing something onto a surface
Throwing something in the air and letting it fall
Putting something surface
Putting something upright on the table
Putting something on flat surface without letting roll
Putting something onto a slanted surface but it doesn't 't glide down
Putting
Putting something into something
Putting something that can't It roll onto a slanted surface, so it slides down
Pretending to put something behind something
Putting something that cannot actually stand upright upright on the table, so it falls on its side
Putting something similar to other things that are already on the table Something falling like feather or paper Something falling like rock
Falling
Bending
Bending something so that deforms
Tipping something over
Tipping
Tipping something with something in it over, something in it falls out
Pretending to close something without actually closing it
Closing
Closing something
Twisting something
Twisting
23

---

Published as a conference paper at ICLR 2026
Pretending trying and failing to twist something
Holding
Holding something
Piling
Piling something up
Showing that something is empty
Showing
Showing something to the camera
Sprinkling
Pretending to sprinkle air onto something
Tearing something just little bit
Tearing
Pretending be tearing something that is not tearable Tearing something into two pieces
Folding
Folding something
Unfolding
Unfolding something
Attaching
Attaching something to something
Tilting something with something it slightly it doesn't fall down
Tilting
Tilting something with something on it until it falls off
Stacking
Stacking number of something
Covering
Covering something with something
Spreading
Pretending to spread air onto something
Dropping
Dropping something onto something
Wiping
Pretending failing to wipe something off of something
NDMD
Cakink W1-022
0.4
Object Scale
0.3
0.2
0.0
0.2
0.3
0.4
0.5
0.6
Scale
Figure 15: The left figure is KDE curves of object scales in different datasets. Our dataset, collected without object category bias, shows more uniform scale distribution,, while the right figure depicts the object size distribution across different verb categories. Actions like "squeezing" typically in- volve smaller objects, whereas "folding" tends to be associated with larger ones.
A.3 DETAILS TOUCH
A.3.1 IMPLEMENTATION DETAILS
Condition Injection Transformer Architecture. Our diffusion model and refiner network fretner are both built upon an 8-layer, 4-head Transformer architecture. This architecture features latent dimension of 512, a feed-forward network size of 1024, the GELU activation function, and a dropout rate of 0.1. To handle the variable number of input points for hands and objects, we designed an Adaptive Feature Selector, which employs a multi-head attention mechanism to aggregate features from the input point clouds into fixed-size representation (64 for the hand and 128 for the object). Text Encoder. To facilitate effective feature alignment and representation learning, we designed and implemented customized encoder architectures tailored different sources of textual features.
24

---

Published as a conference paper at ICLR 2026
Data Collection
Dataset Scale
Dataset Anno.
Dataset
Environmnet
Authentic
Obj Cate.
Obj Inst.
Clips/Prames
Act. Cate.
Contact
HO-3D
lab
10
10
27/78k
none
obman
syn.
8
2.7k
-/154K
none
HO14D lab
16
800
4K/2.4M
54
Oakink* lab/syn. Vix
34/34
100/1.7k
793/230k
5
X
ContactPose lab
25
2.3k/3M
25
GRAB
lab
37
51
1.3k/1.6M
4
X
Ours
wild
403
4.4k
4.4k/-
92
Table 8: Comparison with existing hand-object interaction datasets: HO-3D (Hampali et al.
2020). obman (Hasson et al.) 2019). HOI4D (Liu er al. (022), Oakink* (Yang et al. 2022). Con-
tactPose (Brahmbhatt al. (020), GRAB (Taheri et al. 2020). "Authentic" indicates whether the
hand-object interaction is genuine human behavior. The Oakink* dataset comprises two parts: real
data collected in laboratory settings (img set) and synthetic data (shape set).
For the 4096-dimensional features from the large language model Qwen-7B, we employed a non- linear feature adapter. This module, inspired by established practices in cross-modal alignment, utilizes a Multi-Layer Perceptron (MLP) incorporating GELU activations and Layer Normalization to project the high- dith-smnnniin features into unified 1i2-dimensional latent space. For the 768- dimensional token-level sequence features from models such as BERT and MPNet, we diverged from the conventional mean-pooling strategy. Instead, implemented attention-based pooling mechanism.
Training. The diffusion model is trained for 1000 epochs using the Adam optimizer with learning rate of le-4 and batch size of 128. Subsequently, the refiner network is trained using a distinct set of loss weights tailored for physical plausibility. During this phase, the parameters of the diffusion model are kept frozen.
Refinement. inference time, following a single forward pass through the refiner network, we perform Test-Time Adaptation (TTA) on the generated pose. This process directly optimizes the pose parameters for 500 iterations with learning rate of le-2. The loss weight coefficients used during all training and refinement stages are detailed in Tab.
Table 9: Weight coefficients for each loss term during the training and refinement stages. indicates that the loss term not applicable the corresponding stage.
Parameter
Diffusion Model
Refiner Network
Description
Asimple
1.0
5.0
Base L2 loss
Admmup
0.1
Hand-object distance map loss
Aglohal
0.1
0.1
Hand global pose loss
Pene
100.0
Hand-object penetration loss Hand-object contact loss Cycle-consisteng loss
Agontact
100.0
Xaye
10.0
Xxelf
10000.0
Hand self- self-penttration loss
Aanatomy
0.1
Anatomical plausibility loss
A.3.2 EXPERIMENT SETTINGS
Training Details. We implement model using Accelerate and train it 8 NVIDIA 4090D GPUs with a batch size of 128. We first define 17 detailed hand parts (e.g., pad, nail, knuckle for each finger, palmar, dorsal). Due to the long-tailed distribution of their contact frequencies,
25

---

Published as a conference paper at ICLR 2026
aggregate these parts into semantically coherent categories (pad of each finger, palmar and dorsal). The contact state for any interaction is then encoded as 7-bit binary label, where each bit represents the contact status (1 for contact, 0 for non-contact) of one category. To create balanced training set, we perform resampling based on these unique 7-bit labels.
Evaluation Metrics
Physical Plausibility. We employ three metrics to assess the physical plausibility of the generated hand-object interactions: (1) Mean Vertee Position Error (MPVPE, mm), which computes the average L2 distance between the predicted hand mesh H and the ground truth H; (2) Penetration Depth (PD, cm), measuring the maximum depth of hand vertices penetrating the object surface; (3) Penetration Volume (PV, cm²), quantifying the volumetric intersection by voxelizing the object mesh and calculating the volume within the hand surface.
Contact Accuracy. Contact accuracy is measured by comparing the predicted contact map Cи with the ground truth C₁ using Intersection over Union (IoU) and Fl score.
HOI Diversity. Following (Liu et al. 2023b,, we assess diversity by clustering generated grasps into 20 clusters using K K-means, and report the entropy of cluster assignments and the average cluster size.
Semantic Consistency. Semantic consistency is evaluated by: (1) P-FID, the Fréchet In- ception Distance between point clouds of predicted and ground truth hand meshes, using a pre-trained feature extractor (Nichol et al. 2022); (2) VLM assisted evaluation, where rendered hand-object interactions are scored for semantic alignment with imput captions (0-10 10 scale); (3) Perceptual Score (PS), the mean rating from 10 volunteers (0-10 scale), reflecting the naturalness and semantic consistency of generated grasps.
A.4 USE OF LLMS
We utilized Large Language Models (LLMs) and Vision-Languaa Models (VLMs) to assist in various stages of this research. Specifically, their use can be broken down as follows:
Dataset Annotation: During the dataset generation phase, we employed both LLMS and VLMs automatically generate descriptive text annotations.
Model Architecture: Our proposed model architecture directly incorporates prior knowl- edge from a pre-trained L.LM (Qwen-7B) enhance reasoning and generation capabil- ities.
Evaluation: We used VLM as part of our automated pipeline for the quantitative evalua- tion of model's generated outputs.
Manuseript Preparation: We used LLM- tool for proofreading to help.
26


Reviews:
[{"review_id": "92XwXGDpk5", "reviewer": "Reviewer_JuZb", "summary": "The paper reframes HOI generation from grasp-centric scenarios to free-form interactions and introduces WildO2, an in-the-wild 3D HOI dataset constructed via an O2HOI pairing-and-reconstruction pipeline. It proposes TOUCH, a three-stage framework: (i) text- and geometry-conditioned CVAEs that predict hand/object contact maps; (ii) a multi-stage conditional diffusion model that injects global cues with coarse SSC text early and local geometry with fine DSC text late; and (iii) a lightweight physics-constrained refiner with cycle-consistent contact to correct global pose and sharpen local contacts. The system aims to produce controllable, diverse, and physically plausible interactions beyond grasping.", "questions": "The following questions are based on the weaknesses discussed above; please refer to that section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper contributes a relatively comprehensive dataset to the community and provides useful dataset statistics and analyses in the supp.\n2. The method’s explicit prediction of contact regions, coupled with a coarse-to-fine conditioning schedule, is conceptually sound and well aligned with the goal of improving both global plausibility and local contact fidelity.", "weaknesses": "1. The approach (and the dataset pipeline) relies on one-image-to-3D reconstructions, especially during TTA. Inaccurate object reconstruction can propagate to and bias the estimated hand pose. The paper should analyze or mitigate this dependency—for example, via robustness studies under controlled reconstruction noise, uncertainty-aware weighting, or comparisons with stronger/alternative reconstruction backbones.\n2. While modeling dorsal-side contact is interesting, the paper does not clearly articulate advantages over prior grasp-generation methods such as SemGrasp, which also specifies finger contacts and applies TTA for post-processing. A direct comparison—quantitative and qualitative—under matched prompts and settings would better substantiate the claimed benefits.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TS27beVHEB", "reviewer": "Reviewer_8tRM", "summary": "The paper introduces Free-Form HOI Generation, emphasizing controllable and semantically rich interaction synthesis beyond grasping. In addition, the paper proposes TOUCH, a three-stage framework for text-guided, controllable generation of free-form hand-object interactions (HOI). The multi-level diffusion framework is conditioned on fine-grained text and contact maps, integrating global and local semantic cues for physically plausible synthesis. Finally, the paper introduces WildO2, a large in-the-wild 3D HOI dataset (4.4k interactions, 92 intents, 403 objects) from internet videos via an automated O2HOI reconstruction pipeline. Experiments show the advantage of TOUCH over baselines (ContactGen, Text2HOI) in contact accuracy, plausibility, and semantic alignment.", "questions": "1. How does TOUCH handle ambiguous or conflicting textual intents (e.g., “loosely hold” vs. “grasp tightly”)?\n\n2. What about the generalization to unseen object categories or verbs in WildO2?\n\n3. Could the refinement module be extended to temporal HOI (e.g., multi-frame optimization)?\n\n4. For dataset details, will WildO2 include the intermediate 2D-3D alignment pipeline and failure cases?\n\n5. How does the model behave when the text omits contact information (e.g., only “push the cup”)?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper addresses free-form HOI generation with fine-grained textual control. The three-stage design effectively combines semantics, geometry, and physics.\n\n2. Comprehensive dataset: WildO2 offers unprecedented diversity, with detailed contact annotations and high-quality reconstructions from in-the-wild videos.\n\n3. Strong quantitative and qualitative performance: improvements over state-of-the-art HOI generation baselines across multiple metrics.\n\n4. Clear ablations and insightful analyses: The impact of contact maps, coarse/fine text, and physical consistency is systematically evaluated.", "weaknesses": "1. Static generation limitation: TOUCH focuses on single-frame poses; temporal dynamics (motion continuity, causality) are left for future work.\n\n2. Dataset scale and noise: Although diverse, WildO2 (4.4k samples) remains smaller, and in-the-wild reconstruction errors (≈45% failure rate) suggest potential biases.\n\n3. Comparisons could be expanded: While ContactGen and Text2HOI are solid baselines, comparisons with other text-conditioned 3D diffusion or affordance models (e.g., DiffH2O, Nl2Contact) would strengthen positioning.\n\n4. Ablations on language encoder: The Qwen-7B module shows gains, but results for alternative encoders (e.g., CLIP, BERT) are only briefly summarized. An analysis of semantic faithfulness could be better.\n\n5. Limited discussion on cross-domain generalization: It is unclear how the model generalizes to unseen object categories or out-of-distribution verbs beyond the 92 labeled intents.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6TrcBxtL2p", "reviewer": "Reviewer_TWqj", "summary": "The paper addresses a critical data bottleneck in the hand–object interaction (HOI) domain — the lack of high-quality 3D datasets capturing free-form, non-grasping interactions. While existing datasets focus almost exclusively on structured grasping scenarios collected in laboratory settings, this work proposes WildO2, an in-the-wild 3D HOI dataset that covers diverse everyday manipulations such as pushing, poking, turning, and rotating. WildO2 is automatically constructed from internet videos using an object-only to interaction (O2HOI) frame pairing pipeline, followed by multi-stage 3D reconstruction, contact optimization, and text-based semantic annotation via vision–language models.\n\nBuilding on this dataset, the authors introduce TOUCH, a three-stage text-guided framework for controllable HOI generation. TOUCH integrates (1) explicit contact map prediction, (2) a multi-level conditioned diffusion model that fuses coarse-to-fine text and geometric cues, and (3) a physical refinement module ensuring realistic contact and alignment. Experiments show that TOUCH generates diverse, semantically aligned, and physically plausible free-form interactions, outperforming prior baselines (e.g., ContactGen, Text2HOI) in contact accuracy, plausibility, and diversity metrics.", "questions": "What is the average processing time per frame in the data generation pipeline for WildO2? It would be helpful to know the computational cost and scalability of the proposed reconstruction and alignment procedure.\n\n In the WildO2 dataset, how were the object categories and action types selected? Do the defined free-form action labels correspond to common patterns of real-world human activity, or were they primarily derived from the source video dataset?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper’s WildO2 dataset is a major technical contribution, featuring an well-designed and automated data generation pipeline. This pipeline successfully integrate multi-stage object–hand reconstruction, camera alignment, and physical contact refinement, resulting in high-quality 3D annotations and realistic HOI samples. The inclusion of Descriptive Synthetic Captions (DSCs), generated and verified through vision–language models, is particularly valuable for enabling text-guided interaction synthesis tasks. \n\nThe dataset specifically targets free-form, non-grasping hand–object interactions—a type of everyday manipulation that is pervasive in the real world but consistently overlooked in prior HOI datasets, which mostly emphasize stable grasping or object holding.  the work fills a clear research gap and opens new possibilities for studying intent-driven, semantically controllable HOI generation in both computer vision and embodied AI \n\nthe proposed TOUCH framework (contact → pose → refinement) follows a fairly typical architecture within current interaction synthesis pipelines, it is well-implemented and well-validated through both quantitative and qualitative experiments. Its role here effectively complements the dataset.", "weaknesses": "The WildO2 dataset primarily focuses on rigid objects, while articulated or deformable objects (e.g., clothes, plastic bags, napkins) are absent. These categories are often the most likely to trigger free-form and dynamic hand–object interactions in everyday activities. Although using rigid objects is acceptable for building an initial benchmark, this omission limits the dataset’s ability to fully capture the spectrum of natural, unconstrained human–object interactions.\n\nDespite the paper’s aim to model free-form interactions, the proposed TOUCH framework largely inherits design principles from grasp-based synthesis—treating contact as a quasi-static grasping state. While this formulation is reasonable for static contact modeling, free-form interactions are inherently motion-centric, and thus would benefit from a dynamic or sequence-level synthesis perspective rather than purely static pose generation.\n\nThe use of the term data generation sec 3.2 may be somewhat misleading, as the proposed pipeline mainly performs 3D reconstruction and alignment rather than generative modeling. Although the inclusion of LLM-generated Descriptive Synthetic Captions (DSCs) introduces a generative component, the overall process is better described as a data reconstruction or annotation pipeline to avoid misleading .", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FtQR73100l", "reviewer": "Reviewer_yAMZ", "summary": "This paper tackles free-form HOI generation, pushing the field beyond the standard grasp-centric paradigm. The authors introduce two main contributions: WildO2, a novel 3D HOI dataset reconstructed from in-the-wild videos, and TOUCH, a three-stage diffusion framework designed to synthesize these diverse interactions from fine-grained text prompts. The work is solid, and the results are impressive.", "questions": "1.\tThe reconstruction failure analysis for the dataset is great. Could you also show some typical generation failures of the TOUCH model itself w.r.t. certain objects or text prompts?\n2.\tWhat is the inference speed of the full pipeline? How much overhead does the TTA add, and is it critical for performance?\n3.\tThere is an incomplete citation for Ye et al. (L698) and a missing space in 'WildO2that' (L99).", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1.\tThe formulation of the \"Free-Form HOI\" task is a forward-looking contribution. It moves the community beyond the well-trodden \"grasping\" paradigm toward more realistic and diverse interactions.\n2.\tThe proposed data pipeline for reconstructing 3D HOIs from monocular videos is effective. Creating an in-the-wild 3D dataset like WildO2 is a valuable asset for the community.\n3.\tThe TOUCH framework is technically sound. Its three-stage approach is a logical decomposition of the problem, and the coarse-to-fine conditioning is an effective strategy for fine-grained text control.", "weaknesses": "1.\tThe dataset pipeline's quality is naturally capped by its upstream components (e.g., image-to-3D models, SAM2). Existing 3D generation methods are less used for in-the-wild, low-resolution generation, and a discussion on how to improve the accuracy of the dataset synthesis method in the future would be beneficial.\n2.\tWildO2 excels in interaction diversity. However, its absolute scale is understandably smaller than that of massive lab datasets (e.g., Gigahands). \n3.\tThe method's generalizability needs more validation. Adding experiments on other datasets/domains would strengthen the paper's claims, e.g., qualitative results on OakInk and tests on open-set object or CAD models from sources like Objaverse.\n4.\tThe 4/4 layer split for coarse-to-fine conditioning appears empirical. An ablation study is needed to justify this specific architectural choice against other alternatives (e.g., 2/6).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)