GeneOH Diffusion: Towards Generalizable Hand-Object Interaction Denoising via Denoising Diffusion
Authors: Xueyi Liu
Li Yi
Venue: ICLR 2024 poster (2024)
Keywords: motion refinement
hand-object interaction
inverse problem
generative prior
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: GeneOH Diffusion cleans erroneous out-of-domain HOI tracks with new objects, motions, and novel noise distributions into natural sequences by only training on limited data.

Abstract: In this work, we tackle the challenging problem of denoising hand-object interactions (HOI). Given an erroneous interaction sequence, the objective is to refine the incorrect hand trajectory to remove interaction artifacts for a perceptually realistic sequence.  This challenge involves intricate interaction noise, including unnatural hand poses and incorrect hand-object relations, alongside the necessity for robust generalization to new interactions and diverse noise patterns. We tackle those challenges through a novel approach, GeneOH Diffusion, incorporating two key designs: an innovative contact-centric HOI representation named GeneOH and a new domain-generalizable denoising scheme. The contact-centric representation GeneOH informatively parameterizes the HOI process, facilitating enhanced generalization across various HOI scenarios. The new denoising scheme consists of a canonical denoising model trained to project noisy data samples from a whitened noise space to a clean data manifold and a ``denoising via diffusion'' strategy which can handle input trajectories with various noise patterns by first diffusing them to align with the whitened noise space and cleaning via the canonical denoiser. Extensive experiments on four benchmarks with significant domain variations demonstrate the superior effectiveness of our method. GeneOH Diffusion also shows promise for various downstream applications. We include [a website](https://meowuu7.github.io/GeneOH-Diffusion/) for introducing the work.

Published as a conference paper at ICLR 2024
GENEOH DIFFUSION:
TOWARDS GENERALIZABLE HAND-OBIECT INTERACTION DENOISING VIA DE-
NOISING DIFFUSION
Xueyi Liu Li YP2233
Tsinghua University SSanggaii Al Laboratoryy Shanghai Qi Zhi Institute
Project website: meowuu7, seowuu7.github.. PereGGnneOBDDD Difffiiion
ABSTRACT
this work, we tackle the challenging problem denoising hand-object interac- tions (HOI). Given erroneous interaction sequence, the objective is to refine the incorrect hand trajectory to remove interaction artifacts for perceptually realistic sequence. This challenge involves intricate interaction noise, including unnatural hand poses and incorrect hand-object relations, alongside the necessity for robust generalization to new interactions and diverse noise patterns. We tackle those challenges through a novel approach, GeneOH Diff'usion, incorporating two key designs: an innovative contact-centric HOI representation named GeneOH and a new domain- -amaann-enerrllzabble denoising scheme. The contact-centric representa- tion GeneOH informatively parameterizes the HOI process, facilitating enhanced generalization across various HOI scenarios. The new denoising scheme con- sists of a canonical denoising model trained to project noisy data samples from whitened noise space clean data manifold and "denoising via diffusion" strategy which can handle input trajectories with various noise patterns by first diffusing them to align with the whitened noise space and cleaning via the canon- ical denoiser. Extensive experiments on four benchmarks with significant domain variations demonstrate the superior effectiveness of our method. GeneOH Diffu- sion also shows promise for various downstream applications.
Refining
Rouk
Cleaning motian--ettrgtttll reslts
Bosil
Snapshots from Triining Set
(a) Geseralizable HOT Denoising
(b) Diverse Realts from Stnchvetic Denoising
(6) Applications on Internction Deneising
Figure 1: Trained only on limited data, GeneOH Diffusson can clean novel noisy interactions with new objects, hand motions, and unseen noise patterns (Fig. (a)), produces diverse refined trajectories with discrete manipulation modes (Fig. (b)), and is practical tool for many applications (Fig. (c)).
INTRODUCTION
Interacting with objects is essential part of daily lives, and accurately tracking hands during these interactions has become crucial for various applications, such as gaming, virtual and aug- mented reality, robotics, and human-machine interaction. Yet, this task is highly complex and ill- posed due to numerous factors like intricate dynamics involved and hand-object occlusions. Despite best efforts, existing tracking algorithms often struggle with producing plausible and realistic results.
To better cater to the requirements of downstream tasks, noisy tracking results usually need to be refined. Given hand-object interaction (HOI) sequence with errors, the HOI denoising aims to pro-

---

Published as a conference paper at ICLR 2024
duce natural interaction sequence free of artifacts such penetrations. In this work, we assume the object poses are tracked accurately and focus on refining the hand trajectory following (Zhou et al., 2022; Grady et al., 2021; Zhou al., 2021b; Zhang al., 2021). This setting is impor- tant with many practical demands in applications such as cleaning synthesized motions (Tendulkar et al., 2023; Huang al., 2023; Ghosh et al., 2023; Wu al., 2022), refining motion-retargeted trajectories (Hecker et al., 2008; Tak & Ko, 2005; Aberman et al., 2019), and virtual object manip- ulations (Oh al., 2019; Kato et al., 2000; Shaer al., 2010). Early approaches relied on manually designed priors (Dewaele et al., 2004; Hackenberg et al., 2011), which, however, proved inadequate in handling intricate noise. More recent endeavors have shifted towards learning denoising priors from data (Zhou et al., 2022; 2021b; Grady et al., 2021), yet the existing designs still fall short of providing satisfactory solution.
Leveraging data priors for HOI denoising is challenged by several difficulties. First, the interaction noise highly complex, covering unnatural hand poses, erroneous hand-object spatial relations, and inconsistent hand-object temporal relations. Second, hand movements, hand-object relations, and the noise pattern may vary dramatically across different HOI tracks. For instance, the noise pattern exhibited in hand trajectories estimated from videos differs markedly from that resulted from inaccurate capturing or annotations. A denoising model is often confronted with such out-of-domanl data and is expected to handle them adeptly. However, such a distribution shift poses a substantial challenge for data-driven models. Lacking an effective solution, prior works always cannot clean such complex interaction noise or can hardly generalize to unseen erroneous interactions.
We propose GeneOH Diffusion, a powerful denoising method with strong generalizability and prac- tical applicability (see Figure 1), to tackle the above difficulties. Our method resolves the challenges around two key ideas: 1) designing an effective HOI representation that can both informatively parameterize the interaction and facilitate the generalization by encoding and canonicalizing vital HOI information in coordinate system induced by the interaction region; 2) learning a canonical denoiser that projects noisy data from a whitened noise space to the data manifold for domain- generalizable denoising. A satisfactory representation that parameterizes the high-dimensional HOI process for denoising should be able to represent the interaction process faithfully. highlight noises, and align different HOI tracks well to enhance generalization capabilities Therefore, we introduce GeneOH, Generalized contact-centric Hand-Object spatial and temporal relations. GeneOH en- codes the interaction informatively, encompassing the hand trajectory, hand-object spatial relations, and hand-object temporal relations. Furthermore, it adopts a contact-centre perspective and in- corporates innovative canonicalization strategy. This approach effectively reduces disparities between different sequences, promoting generalization across diverse HOI scenarios. To enhance the denoising model's generalization ability to novel noise distributions, our second effort centers on the denoising scheme side. We propose to learn canonical denoising model that describes the map- ping from whitened noise space to the data manifold. The whitened noise space contains noisy data diffused from clean data in the training dataset via Gaussian noise at various noise scales. With the canonical denoiser, we then leverage a "denoising via diffusion" strategy to handle input trajectories with various noise patterns in a domain-generalizzbbe manner. It first aligns the input to the whitened noise space by diffusing it via Gaussian noise. Subsequently, the diffused sample is cleaned by the canonical denoising model. To strike balance between the denoising model's generalization pability and the faithfulness of the denoised trajectory, we introduce a aapper-rrrmmter that decides the scale of noise added during the diffusion process, ensuring the diffused sample remains faithful to the original input. Furthermore, instead of learning to clean the interaction noise through a single stage, we devise progressive denoising strategy where the input is sequentially refined via three stages, each of which concentrates on cleaning one specific component of GeneOH.
We conduct extensive experiments on three datasets, GRAB (Taheri et al., 2020), a high-quality MoCap dataset, HO14D (Liu et al., 2022), a real-world interaction dataset with noise resulting from inaccurate depth sensing and imprecise vision estimations, and ARCTIC (Fan et al., 2023). a dataset featuring dynamic motions and changing contacts, showing the remarkable effectiveness and gener- alizability of our method. When only trained on GRAB, our denoiser can generalize to HO14D with novel and difficult noise patterns and ARCTIC with challenging interactions, surpassing prior arts by a significant margin, as demonstrated by the comprehensive quantitative and qualitative compar- isons. We will release our code to support future research. In summary, our contributions include:
An HOI denoising framework with powerful spatial and temporal denoising capability and unprecedented generalizability to novel HOI scenarios;
2

---

Published as a conference paper at ICLR 2024
An HOI representation named GeneOH that can faithfully capture the HOI process, high- light unnatural artifacts, and align HOI tracks across different objects and interactions;
An effective and domain- genariiigenealizablee denoising method that can both generalize across different noise patterns and clean complex noise through progressive denoising strategy.
RELATED WORKS
Hand-object interaction is an important topic for understanding human behaviors. Prior works wards this direction mainly focus on data collection (Taheri et al., 2020; Hampali et al., 2020; Guzov Fan et al., 2023; Kwon al., 2021), reconstruction (Tiwari al., 2022; Xie al., 2022; al., 2022;
Qu et al., 2023; Ye et al., 2023), interaction generation (Wu et al., 2022; Tendulkar et al., 2023; Zhang & Tang, 2022; Ghosh et al., 2023; et al., 2023), and motion refinement (Zhou et al., 2022:
Grady et al., 2021; Zhou et al., 2021b; Núñez, 2022). The HOI denoising task wishes to remove unnatural phenomena from HOI sequences with interaction noise. In real application scenarios, a denoising model would frequently encounter -ft-o---mmmin interactions, and is expected to gener- alize to them. This problem is then related to domain generalization, general machine learning topic (Sicilia et al., 2023; Segu et al., 2023; Wang et al., 2023; Zhang et al., 2023; Jiang al., 2022; Wang et al., 2022; Blanchard et al., 2011; Muandet et al., 2013; Dou et al., 2019), where a wide range of solutions have been proposed in the literature. Among them, leveraging domain invariance to solve the problem is promising solution. Our work is related to this kind of approach, at a high level. However, what the domain invariant information for the HOI denoising task, and how to encourage the model to leverage such information for denoising remains very tricky. We focus on designing invariant representations and learning a canonical denoiser for Noomain-generllzzan noising. Moreover, we are also related to intriguing works that wish to leverage data priors to solve the inverse problem (Song et al., 2023; Mardani et al., 2023; Tumanyan et al., 2023; Meng et al., 2021; Chung et al., 2022). For task, we need to answer some fundamental questions regarding what are generalizable denoising priors, how to learn them from data, and how to leverage the prior to refine noisy input from different distributions. We'll illustrate our solution in the method section.
de-
HAND-OBJECT INTERACTION DENOISING VIA DENOISING DIFFUSION
Given an erroneous hajd-object interaction sequence with K frames (H, 0) [H,g Oz) we assume the object pose trajectory {O+] accurate following (Zhou al., 2022: 2021b; Grady et al., 2021; Zhang et al., 2021) and aim at cleaning the noisy hand trajectory { This set- ting is of considerable importance, given its practical applicability in various domains (Tendulkar et al., 2023; Ghosh et al., 2023; Li et al., 2023; Wu et al., 2022; Hecker et al., 2008; Oh et al., 2019; Shaer et al., 2010). The cleaned hand trajectory should be free of unnatural hand poses, in- correct spatial penetrations, and inconsistent temporal hand-object relations. The hand trajectory should present visually consistent motions and adequate contact with the object support manip- ulation. The problem is ill-posed in nature owing to the difficulties posed by complex interaction noise and the substantial domain gap across different interactions resulting from new objects, hand movements, and unseen noise patterns.
We resolve the above difficulties by 1) designing a novel HOI representation that parameterizes the HOI process faithfully and can both simplify the distribution of complex HOI and foster the model generalization across different interactions (Section 3.1) and 2) devising an effective de- noising scheme that can both clean complex noises through progressive denoising strategy and generalize across different input noise patterns (Section 3.2).
3.1 GENEOH GENERALIZED CONTACT-CENTRIC HAND-OBJECT SPATIAL AND TEMPORAL RELATIONS
Designing an effective and generalizable HOI denoising model requires serious effort in the repre- sentation design. It involves striking a balance between expressive modeling of the interaction with objects and supporting the model's generalization to new objects and interactions. The ideal HOI representation should accurately capture the interaction process, highlight any unusual phenomena like spatial penetrations, and facilitate alignment across diverse interaction sequences.
3

---

Published as a conference paper at ICLR 2024
We introduce GeneOH to achieve this. It integrates the hand trajectory, hand-object spatial relations, and hand-object temporal relations to represent the HOI process faithfully. An effective normal- ization strategy is further introduced to enhance alignment across diverse interactions. The hand trajectory and the object trajectory are compactly represented as the trajectory of hand keypoints, denoted as J {J&) k= and the interaction region sequence: P {Px) in contact-aware manner. We will then detail the design of GeneOH.
Generalized contact points. The interaction region is established based on points sampled from the ob- ject surface close to the hand trajectory. referred to as "generalized contact points". They are No points (denoted as P E TR. No3) sampled from object sur- face points, whose distance to the hand trajectory does not exceed a threshold value of r. (set to 5mm). The sequence of these points across all frames is rep- resented by P = { where Pa denotes points at frame k. Each Pμ is associated with 6D pose, consisting of the object's 's orientation (or the orienta- tion of the first part for articulated objects), denoted as Rg, and the center of PR. denoted as fy-
GoneOH Cljor Bulato Skeiig Divalian
Haad
Reiatie Pastica N2W
Mavey Thrmiiue
Nammal
to Exneiiclted lland Trajewsyy
Che Conrae Counts lied Otjer Speai Balition
Prams
Domek
Baad-Cheed Ralativ Valecitar
Object
60) Geneeiiiee Contur
Objert Teepeed Noties-vere Rokaioon
Figure 2:
Three components of GeneOH.
Canonicalized hand trajectories. We include hand trajectories in representation to effectively model hand movements. Specifically, we leverage hand keypoints to model the hand, as they offer a compact and expressive representation, We represent the hand trajectory as the sequence of 21 hand keypoints, denoted as J = {Je E R Jk-1- where N, 21. We further canonicalize the hand trajectory J using the poses of the generalized contact points to eliminate the influence of object poses, resulting in the canonicalized hand trajectory in GeneOH : J = {Jx = (J, tx R 1==1
Generalized contact-centric hand-object spatial relations. We further introduce hand-object spatial representation in GeneOH. The representation is based on hand keypoints and general- ized contact points to inherit their merits. The spatial relation centered at each generalized con- tact point оş E Ps. comprises the relative offset from Oß to each hand keypoint h& JR, i.e., (ha- - oy lh/ E Ju the object point normal ng. and the object point position Oμ- These statis- ties are subsequently canonicalized using the 6D pose of the generalized contact points to encour- age cross-interaction alignment. Formally, the spatial representation centered at o₁ is defined as: $g = ((og- t. R nę RT, {(h 0g R hh J). The spatial relation S is composed of se at each generalized contact point: S {( og E P By encoding object normals and hand- object relative offsets, S can reveal unnatural hand-object spatial relations such as penetrations.
Generalized contact-centric hand-object temporal relations. Considering the limitations of the above two representations in revealing temporal errors such as incorrect manipulations resulting from inconsistent hand-object motions, further introduce hand-object temporal relations to pa- rameterize the HOI temporal information explicitly. We again take hand keypoints J to represent hand shape and generalized contact points P for the object shape take advantage of their good ability in supporting generalization. The temporal relations encode the relative velocity between each hand point où and each hand keypoint h2 at frame (v- v vf). the Euclidean distance between each pair of points (d l|h owll2), and the object velocity vß in the representation, as illustrated in Figure 2. We further introduce two statistics by using the object point normal to canon- icalize v resulting in two normalized statistics: ho . orthogonal to the object tangent plane, and ho "k."" lying in the object's tangent plane, and encoding them with hand-object relative distances: ho = = ko RR
k.j [|2- Here, k. ka, and kis are positive hyper- R.L. ||2 and cx] e k- k. w
parameters, and the term e d is negatively related to the distance between the hand and object points. This canonicalization and encoding strategy aims to encourage the model to learn different denoising strategies for the two types of relative velocities, enhance cross-interaction generalization by factoring out object poses, and emphasize the relative movement between very close hand-object point pairs. The temporal representation T is defined by combining the above statisties of each hand-object point pair across all frames together:
ho T = [vv {aho ho 1 E [hg E Jo E K-1
(I)
It reveals temporal errors by encoding object velocities, hand-object distances and relative velocities.
4

---

Published as a conference paper at ICLR 2024
Pregrosine HOT Denoising
Spracbet 19
Makes kil
Denoising via Diflusion
Sanrik
Diffae
Di Clas ri Difaiia
Dewdte Urarner Devie:
Dat
Whirend
Chrpur Enjectery
Space
Diflus
Dexiic
Dariiation
The progressive HOI denoosing gradually cleans the input noisy trajectory through three stages. Each stage concentrates on refining the trajectory by denoising specific part of GeneOH via a canonical denoiser through the "denoising via diffusion" strategy.
Figure 3:
The GeneOH representation. The overall representation, GeneOH, comprises the above three components, as defined formally: GeneOH = (J.S, 7.. Figure illustrates the design. It faithfully captures the interaction process, can reveal noise by encoding corresponding statistics, and benefits the generalization by employing carefully designed canonicalization strategies. Inspecting back into previous works, TOCH (Zhou al., 2022) does not explicitly parameterize the hand-object temporal relations or hand shapes and does not carefully consider the spatial canonicalization to facilitate the generalization, which limits its denoising capability and may lead to the loss of high-frequency hand pose details. ManipNet (Zhang et al., 2021) does not encode temporal relations and does not incorporate contact-centric canonicalization, rendering it inadequate for capturing the interaction process and less effective for generalization purposes.
3.2 GENEOH DIFFUSION: PROGRESSIVE HOI DENOISING VIA DENOISING DIFFUSION
While GeneOH excels in encoding the interaction process faithfully, highlighting errors to facilitate denoising, and reducing the disparities among various interaction sequences, designing an effective denoising model is still challenged by complex interaction noise, even from distribution unseen during training. Previous methods typically employ patter--spece map noisy data restricted to certain patterns to the clean data manifold (Zhou et al., 2022; 2021b). However, these methods are susceptible to overfitting, resulting in conceptually incorrect results when faced with interactions with unseen noise patterns, evidenced in our experiments.
denoising models trained to
To ease the challenge posed by novel in- teraction noise, we propose new denois- ing paradigm that learns a canonical de- noising model and leverages for domain- generalizable denoising. It describes the map- ping from noisy data at various noise scales from a whitened noise space to the data man- ifold. The whitened noise space populated with noisy data samples diffused from the clean data via a diffusion process which gradually adds Gaussian noise the data according to variance schedule, a similar flavor to the for- ward diffusion process in diffusion-based generative models (Song al., 2020; Ho al., 2020; Rombach et al., 2022; Dhariwal & Nichol, 2021). With the canonical denoiser, we then leverage a "denoising via diffusion" strategy handle input trajectories with various noise patterns in a gen- eralizable manner. It first diffuses the input trajectory & via the diffusion process to another sample ã that resides closer to the whitened noise space. Then the model projects the diffused sample to the data manifold. To balance the generalization ability of the denoising and the fidelity of the denoised result the input, the diffused needs to be faithful to the input å. We then introduce diffusion timestep lair that decides how many diffusion steps are added. The process is visually depicted the right part of Figure: 3. Details are outlined in Algorithm 1. We also implement the gen-
Algorithm 1 Denoising via Diffusion
Input: forward diffusion function Diffuse(-, t), the de- noising model (enoiso t), input noisy point &, diffusion steps tair.
Output: denoised data
I: function DENOISH(a/Aur, fan)
for from fair to
3:
z ~ denoise
4: return
5: & DDiffuse(&, tam)
6: return + DENOISE(E, tair)
denoising model's function and the training those of the score functions in diffusion-baase crative models. It is a multi-step stochastic denoiser that eliminates the noise of the input gradually to step-by-step. This way the denoiser can deal with noise at different scales fiexibly and can give multiple solutions for the ill- posed ambiguous denoising problem.
Based on the domain-gent generalizable denoising strategy, designing single data-driven model to clean heterogeneous interaction noise in one stage is still not feasible. The interaction noise contains various kinds of noise at ununiform scales stemming from different reasons. Thus the corresponding
5

---

Published as a conference paper at ICLR 2024
noise-to-date mapping is very high dimensional and is very challenging learn from limited data. A promising solution to tackle the complexity is taking a progressive approach and learning multiple specialists, each concentrating stage formulation brings new difficulties. It necessitates careful consideration of the information to be cleaned at each stage prevent the current stage from compromising the naturalness achieved in previous stages. Fortunately, our design of the GeneOH representation facilitates a solution to this issue. HOI information can be represented into three relatively homogeneous parts: J. S, and T. Furthermore, their relations ensure the sequential refinement of the hand trajectory by denoising its J. S. and T representations across three stages can avoid the undermining problem. A formal proof of this property is provided in the Appendix A.2.
cleaning specific type of noisy information. However, the multi-
Progressive HOI denoising. We design a three-stage denoising approach (outlined in Figure 3), each stage dedicated to cleaning one aspect of the representation: J, S, and T, respectively. In each stage, a canonical denoising model is learned for the corresponding representation, and the denoising is carried out using the "denoising via diffuusion" strategy. Given the input GeneOH"puu
{fimput Sinput i j, the first denoising stage, named MotionDiff, denoises the noisy canonical
hand trajectory Finpat to Jstuge, One denoised hand trajectory J can be easily computed by de-canonicalizing Jssaae; using object poses. GeneOHimpu Iupur can also be updated accordingly into }. Then the second stage, named SpatialDiff, denoises the noisy spatial relation Sauae, to Sttges, Two stages-denoised hand trajectory Jaugs= can be trans- formed from the hand-object relative offsets in Sauggi; Jstage = (ha
GeneOHaange = { Sruage Frange, S
(og E Pk)- Following this, GeneOHsinge, will be updated to GeneOH = { Sutage, Juages Finally the last stage. named TemporalDifr, denoises Frange to T Since temporal information such as relative velocities is redundantly encoded in T. we compute the three stages-denoiset hand tra- jectory Jstages by optimizing J so that its induced temporal representation aligns with Tstages And we take Jsinge as the final denoising output, denoted as J. Each stage would not undermine the naturalness achieved after the previous stages, proved in the Appendix A.2.
Fitting for a hand mesh trajectory. With the denoised trajectory J and the object trajectory, a parameterized hand sequence represented via MANO parameters {rk, Lx 3e, Og ) are optimized to fit J well. Details are illustrated in the Appendix A.3.
EXPERIMENTS
We conduct extensive experiments demonstrate the effectiveness of our method. We train all models on the same training dataset and introduce four four test sets with different levels of domain shift assess their denoising ability and the generalization ability (see Section 4.2). Moreover, we demonstrate the ability of our denoising method to produce multiple reasonable solutions for a single input in Section 4.3, At last, we show various applications that we can support (Section 4.4). Another series of experiments using a different training set is presented in the Appendix B.I.
4.1 EXPERIMENTAL SETTINGS
Training datasets. All models are trained on the GRAB dataset (Taheri al., 2020). We follow the cross-object splitting strategy used in TOCH (Zhou et al., 2022) and train models on the training set. Our denoising model only requires ground-trume sequences for training. For those where the noisy counterparts are demanded, we perturb each sequence by adding Gaussian noise on the hand MANO translation, rotation, and pose parameters with standard deviations set to 0.01, 0. 0.5 respectively.
Evaluation datasets. We evaluate our model and baselines on four distinct test sets, namely GRAB test set with Gaussian noise, GRAB (Beta) test set with noise sampled from Beta distribution (8(8,2) 2)), HOI4D dataset (Liu et al., 2022) with real noise patterns resulting from depth sensing errors and inaccurate pose estimation algorithms, and ARCTIC dataset (Fan et al., 2023) with Gaus- sian noise but containing challenging bimanual and dynamic interactions with changing contacts. Noisy trajectories with synthetic noise are created by adding noise sampled from corresponding distributions the MANO parameters.
Metrics. We introduce two sets of evaluation metrics. The first set focuses on assessing the model's ability to recover GT trajectories from noisy inputs following previous works (Zhou et al., 2022),

---

Published as a conference paper at ICLR 2024
including Mean Per-Joint/Vertex Position Error (MPJPE/MPVPE), measuring the average distance between the denoised hand joints or vertices and the corresponding GT positions and Contact loU (C-IoU) assessing the similarity between the contact map induced by denoised trajectory and the GT. The second set quantifies the quality of denoised results, including Solid Intersection Volume (IV) and Penetration Depth, measuring penetrations, Proximity Error, evaluating the difference of the hand-object proximity between the denoised trajectory and the GT, and HO Motion Consistency, as- sessing the hand-object motion consistency. Detailed calculations are presented in the Appendix C.2.
GRAB (thin geometry)
GRAB (Beta) (novel synthetic noise)
HOL4D (novel real noise, thin geometry)
Nolyy Input
TOCH
Oax
Timestamp
Timestamp
Timestamp
ARCTIC (novel articulated motions, changing contacts)
Notey input
Oues
Teack and love
Touns th
Open chas the ld
Timastamp
Figure 4: Qualitative comparisons. different ways resolve penetrations
Please refer to our website and video for animated results.
divvesse grasping poses
different manipulation strategies
Result 2 Resul1 Noisy Insut
Timestamp
Timestamp
Timestamp
Figure Stochastic denoising can produce divserse results with discrete modes.
Baselines. We compare model with the prior art on the HOI denoising problem, TOCH (Zhou et al., 2022). A variant named "TOCH (w/ MixStyle)" is further created by combining TOCH with general domain generalization method MixStyle (Zhou et al., 2021a). Another variant, "TOCH (w/ Aug.)", where TOCH is trained on the training sets of the GRAB and GRAB (Beta), is further introduced enhance its robustness towards unseen noise patterns.
Evaluation settings. When evaluating our model, we select the trajectory that closest to the imput noisy trajectory from 100 randomly sampled denoised trajectories using seeds from 0 to 99. For deterministic denoising models, we report the performance on a single run. Since our model can give multiple solutions for single input, we additionally report the performance of our model in the form of average with standard deviations in the Appendix on the second metric set measuring quality.
4.2 HOI DENOISING
We evaluated our model and compared it with previous works on four test sets: GRAB, GRAB (Beta), HOI4D, and ARCTIC. In the GRAB test set, all objects were unseen during training, re- sulting in shift in the interaction distribution. In the GRAB (Beta) test set, the object shapes, interaction patterns, and noise patterns differ from those in the training set. The H014D dataset includes interaction sequences with novel objects and unobserved interactions, along with real noise
7

---

Published as a conference paper at ICLR 2024
Table 1:
Quantitative evaluations and comparisons baselines.
Bold red numbers for best
values and italic blue values for the second best-performed ones. "GT" stands for "Ground-Trutt""
MPIPE
MPVPE
C-JoU
IV
Penctration Depth
Proximity Ener (vem,
Motion Consistency
Dutaset
Method
(own,
(min,
(%,
(ome,
(man,
(mam
GT
0,50
1.33
Input
23.16
22.78
1.01
4.48
5.25
13.29
881.23
GRAB
TOCH
12.38
2,09
2.17
TOCH (w/ MixStyle)
13.36
13.03
23.20
2.28
2.62
3,10
21.29
TOCH (w/ Aug.)
12.23
11.89
22.71
1.94
2.04
3.16
22.58
Ours
9.22
25.27
1.23
1.74
Inpot
17.65
17:40
13.21
2.19
4.77
5.83
27,58
GRAB
(Beta)
TOCH (w/ MixStyle)
22.79
21.19
16.28
2.01
2,63
4.65
17.37
TOCH (w/ Aug.)
11.65
10.47
24.87
1.52
1.86
3.07
13.09
Ours
9.09
8.98
26.76
1.19
1.69
2.74
0.52
Input
2.26
2:47
46.45
HOHD
TOCH (w/ MixSiyle)
4.31
4.96
25.67
TOCH Ang.)
4.20
4.51
25.85
Ouns
1.99
2.15
GT
0.33
0.92
0.41
Input
25.51
24.84
1.68
2.28
4.89
15:21
931.69
ARCTIC
TOCH
14.34
14.07
20.32
1.84
2.01
4.31
18.50
MixStyle)
TOCH (w/ Aug.)
14.18
13.90
20.10
1.75
1.98
5.64
22.57
Ours
11.57
11.09
23.49
1.35
1.93
2.71
0.92
caused by inaccurate sensing and vision estimations. The ARCTIC dataset contains challenging bimanual dexterous HOI sequences with dynamic contacts. Table and Figure 4, summarize the quantitative results and can demonstrate the superiority of our method recover GT sequences and produce high-quality results compared to previous baseline methods. We include more results in the Appendix B.1, our website and video.
Performance on challenging noisy interactions. As shown in Figure 4, the perturbed noisy trajec- tories exhibit obvious problems such unnatural hand poses, large and difficult penetrations such as penetrating the thin mug handle, and unrealistic manipulations caused by incorrect contacts and inconsistent hand-ohject motions. Our method can produce visually appealing interaction sequenees from noisy inputs effectivvll. Besides, do not have difficulty in handling difficult shapes such as the mug handle and scissor rings which are very easy to penetrate. However, TOCH cannot perform well. Its results still exhibit obvious penetrations (the last frame) and hand motions that are insuffi- cient to manipulate the mug. Furthermore, we are not challenged by difficult and dynamic motions with changing contacts, as demonstrated by results on the ARCTIC dataset.
Results on noisy interactions with unseen noise patterns. Figure 4, we demonstrate our method's robustness against new noise patterns, including previously unseen synthetic noise and novel real noise. Our approach effectively cleans such noise, producing visually appealing and motion-aware results with accurate contacts. In contrast, TOCH fails in these scenarios, as ex- hibits obvious penetrations (as seen in the middle example) and results in stiff hand trajectories without proper contacts to manipulate the object (as seen the rightmost example).
4.3 STOCHASTIC HOI DENOISING
Figure 5 illustrates our ability provide multiple plausible denoised results for single noisy input. Notably, we observe discrete manipulation modes among these results. For instance, in the leftmost example of Figure 5, our model generates different hand poses to address the unnatural phenomenon in the second frame, where two fingers penetrate through the camera. Similarly, in the rightmost example, our results offer distinct ways rotate the scissor for certain angle.
4.4 APPLICATIONS
Cleaning hand trajectory estimations. As denoising model, approach effectively refine hand trajectory estimations derived from image sequence observations. Figure 6 provides examples of applying our model to estimations obtained from AretieNet-L.STM (Fan al., 2023).
8

---

Published as a conference paper at ICLR 2024
Refining noisy retargeted hand motions. In the right part of Figure 6, we showcase the application of our denoising model in cleaning noisy retargeted hand trajectories. Our model excels at resolving penetrations present in the sequence resulting from direct retargeting. In contrast, TOCH's result still suffers from noticeable penetrations.
ArraNaa
LSTX
Clowinn UKt attration
Stenrs
Cleriing injectrris muviin and direclly retagated matize)
Clicsina edargeze basi injotiic
Cure Denibel
Tinesarip
Figure 6: Applications refining noisy hand trajectories estimated from videos (left) and cleaning retargeted hand trajectories (right).
ABLATION STUDY
Generalized contact-centric parameterizations. GeneOH leverages generalized contact points normalize the hand-object relations. To assess the effectiveness of this design, We create an ablated model named "Ours (w/o Canon.)", which uses points sampled from the entire object surface for pa- rameterizing. From Table 2, we can observe that our design on parameterizing around the interaction region can successfully improve the model's s generalization ability towards unseen interactions.
Denoising via diffusion. To further investi- gate the impact of the "denoising via diffusion" strategy on enhancing the model's generaliza- tion ability, we ablate it by replacing the denois- ing model with an autocncoder structure. The results are summarized in Table 2. Besides, the comparisons between "Ours (w/o Diffusion)" and TOCH highlight the superiority of our representation GeneOH as well.
Table 2: Ablation studies on the HO14D dataset.
Methad
(em! D>
IV
Penctiation Depth
Grom, 1)
HO Motion Consistency (mar" J)
Input
2.26
2.47
46.45
Dar (wis Specialoor
2.04
145
31.67
Ours (wl TemporalDith
1.72
1.90
34,25
Our (wlo Diffusion)
3.36
3.83
15.65
Ours (wio Canon) Ours
2:36
3:57
13.26
1.99
275
9,81
Hand-object spatial and temporal denoising. We propose a progressive denoising strategy composed of three stages to clean the complex interaction noise. This multi-stage ap- proach is crucial, as a single denoising stage would fail to pro- duce reasonable results in the presence of complex interaction noise. To validate the effectiveness of the stage-wise denois- ing, we created two ablated versions: a) "Ours (w/o Tempo- ralDiff)" by removing the temporal denoising module, and b) "Ours (w/o SpatialDiff)" by removing both the temporal and spatial denoising modules. Figure 7 and Table 2 demonstrate their effectiveness removing unnatural hand-object penetra- tions and enforcing consistent hand-object motions.
Oae Oieeoo Gen
Figure 7: Effectiveness of the Spa- tialDiff and TemporalDiff stages.
More quantitative and qualitative results for ablation studies are included the Appendix B.2.
CONCLUSION AND LIMITATIONS
In this work, propose GeneOH Diffusion to tackle the generalizable HOI denoising problem. We resolve the challenge by 1) designing an informative HOI representation that is friendly for generalization, and 2) learning canonical denoising model for Domainngeeneallzann denoising. Experiments demonstrate our high denoising capability and generalization ability.
Limitations. The main limitation lies in the assumption of accurate object pose trajectories. It may not hold if the HOI sequences are estimated from in-the-wild videos. Refining object poses and hand poses the same time is valuable and practical research direction.

---

Published as a conference paper at ICLR 2024
REFERENCES
Kfir Aberman, Rundi Wu, Dani Lischinski, Baoquan Chen, and Daniel Cohen-Or. Learning character-agnostic motion for motion retargeting in 2d. arXiv preprint arXiv: /ttp//955500800/ 2019.
Gilles Blanchard, Gyemin Lee, and Clayton Scott. Generalizing from several related classification tasks to a new unlabeled sample. Advances in neural information processing systems, 24, 2011.
Hyungjin Chung, Byeongsu Sim, Dohoon Ryu, and Jong Chul Ye. Improving diffusion models for inverse problems using manifold constraints. arXiv preprint arXiv:2206.0094 2022. 3
Guillaume Dewaele, Frédéric Devernay, and Radu Horaud. Hand motion from 3d point trajectories and a smooth surface model. In European Conference on Computer Vision, pp. 495-507. Springer, 2004. 2
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis, Advances in Neural Information Processing Systems, 34:8780-8794, 2021. 5
Qi Dou, Daniel Coelho de Castro, Konstantinos Kamnitsas, and Ben Glocker. Domain generaliza- tion via Systems, 32, 2019.
learning of semantic features. Advances in Neural Information Processing
Zicong Fan, Omid Taheri, Dimitrios Tzionas, Muhammed Kocabas, Manuel Kaufmann, Michael J. Black, and Otmar Hilliges. ARCTIC: A dataset for dexterous bimanual hand-object manipulation. In Proceedings IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2023. 2,
3, 8, 31
Anindita Ghosh, Rishabh Dabral, Vladislav Golyanik, Christian Theobalt, and Philipp Slusallek. Imos: Intent-driven full-body motion synthesis for human-object interactions. In Computer Graphics Forum, volume 42, pp. 1-12. Wiley Online Library, 2023. 2,
Patrick Grady, Chengeheng Tang, Christopher D Twigg, Minh Vo, Samarth Brahmbhatt, and Charles C Kemp. Contactopt: Optimizing contact to improve grasps. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1471-1481, 2021. 2,
Vladimir Guzov, Torsten Sattler, and Gerard Pons-Moll. Visually plausible human-object interaction capture from wearable sensors. arXiv preprint 2-Xxi:2205.0288:0 2022. 3
Georg Hackenberg. Rod McCall, and Wolfgang Broll. Lightweight palm and finger tracking for real-time 3d gesture control. In 2011 IEEE Virtual Reality Conference, pp. 19-26. IEEE, 2011. 2
Shreyas Hampali, Mahdi Rad, Markus Oberweger, and Vincent Lepetit. Honnotate: A method for 3d annotation of hand and object poses. In CVPR, 2020. 3
Chris Hecker, Bernd Raabe, Ryan W Enslow, John DeWeese, Jordan Maynard, and Kees van Prooi- jen. Real-time motion retargeting to highly varied user-created morphologies. ACM Transactions Graphics (TOG), 27(3)1-11, 2008. 3
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 68:5140-6851, 2020. 5
Siyuan Huang, Zan Wang, Puhao Li, Baoxiong Jia, Tengyu Liu, Yixin Zhu, Wei Liang, and Song- Chun Zhu. Diffusion-based generation, optimization. and planning in 3d scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16750-16761, 2023. 2
Junguang Jiang, Yang Shu, Jianmin Wang, and Mingsheng Long. Transferability in deep learning: A survey. arXiv preprint arXiv:2201.0Se8 2022.
Hirokazu Kato, Mark Billinghurst, Ivan Poupyrev, Kenji Imamoto, and Keihachiro Tachibana. Vir- tual object manipulation on a table-top ar environment. In Proceedings IEEE and ACM Interna- fional Symposium on Augmented Reality (ISAR 2000), pp. 111-119. leee, 2000. 2
10

---

Published as a conference paper at ICLR 2024
Taein Kwon, Bugra Tekin, Jan Stühmer, Federica Bogo, and Mare Pollefeys. H2o: Two hands manipulating objects for first person interaction recognition. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 20138-10148, 2021. 3
Quanzhou Li, Jingbo Wang. Chen Change Loy, and Bo Dai. Task-oriented human- interactions generation with implicit neural representations. arXiv preprint arXiv:2303.13119, 2023. 3
Yunze Liu, Yun Liu, Che Jiang, Kangbo Lyu, Weikang Wan, Hao Shen, Boqiang Liang, Zhoujie Fu, He Wang, and Li Yi. Hoi4d: A 4d egocentric dataset for category-level human-object interaction. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp- 21013 21022,, 2022. 2, 6,
Morteza Mardani, Jiaming Song, Jan Kautz, and Arash Vahdat. A variational perspective on solving inverse problems with diffusion models. arXiv preprint arXiv:2305.04331, 2023. 3
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations, 2021. 3
Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain generalization via invariant feature representation. In International conference on machine learning, pp. 10-18. PMLR, 2013.
Johnny Núñez. Comparison of Spatio-Temporal Hand Pose Denoising Models. PhD thesis, Univer- sitat DE Barcelona, 2022. 3
Ju Young Oh, Ji Hyung Park, and Jung-Min Park. Virtual object manipulation by combining touch and head interactions for mobile augmented reality. Applied Sciences, 914):2993, 2019. 2,
Wentian Qu, Zhaopeng Cui, Yinda Zhang, Chenyu Meng, Cuixia Ma, Xiaoming Deng, and Hongan Wang. Novel-view synthesis and pose estimation for hand-object interaction from sparse views. arXiv preprint arXiv:2308. /98, 2023. 3
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Javier Romero, Dimitrios Tzionas, and Michael J Black. Embodied hands: Modeling and capturing hands and bodies together. arXiv preprint arXiv:2201. 022600, 2022. 18
Mattia Segu, Alessio Tonioni, and Federico Tombari. Batch normalization embeddings for deep domain generalization. Puttern Recognition, 109115, 2023. 3
Orit Shaer, Eva Hornecker, et al. Tangible user interfaces: past, present, and future directions. Foundations and Trends@ in Hluman-Copputer Interaction, 3(1-2):4-377, 2010. 2, 3
Anthony Sicilia, Xingchen Zhao, and Seong Jae Hwang. Domain adversarial neural networks for domain generalization: When it works and how to improve. Machine Learning, pp. 1-37, 2023.
3
Bowen Song, Soo Min Kwon, Zecheng Zhang, Xinyu Hu, Qing Qu, and Liyue Shen. Solv- ing inverse problems with latent diffusion models via hard data consistency. arXiv preprint arXiv:2307.08123, 2023. 3
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456. 2020. 5
Omid Taheri, Nima Ghorbani, Michael Black, and Dimitrios Tzionas. Grab: A dataset of whole- body human grasping of objects. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IV 16, pp. 581-600 Springer, 2020. 2, 3, 6, 30
11

---

Published as a conference paper at ICLR 2024
Seyoon Tak and Hyeong- -Sok Ko. A physically- basedd motion retargeting filter. ACM Transactions on Graphics (TOG),
117, 2005. 2
Purva Tendulkar, Didac Surís, and Carl Vondrick. Flex: grasps. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 21179-21188 2023. 2,
Full-body grasping without full-body
Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen-Or, and Amit H Bermano. Human motion diffusion model. arXiv preprint arXiv:2209.14911 14916, 2022. 32
Garvita Tiwari, Dimitrije Antić, Jan Eric Lenssen, Nikolaos Sarafianos, Tony Tung, and Gerard Pons-Moll Pose-ndf: Modeling human pose manifolds with neural distance fields. In Computer Aision-ECCy 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceed- ings, Part V. pp. 572-552-- 179-589.005 Springer, 2022. 3
Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven tonge-to-image translation. In Proceedings of the IEEE/CVF Conference Com- puter Vision and Pattern Recognition, pp. 1921-1930, 2023. 3
Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Tao Qin, Wang Lu, Yiqiang Chen, Wenjun Zeng, and Philip Yu. Generalizing to unseen domains: survey on domain generalization. IEEE Transactions on Knowledge and Data Engineering, 2022. 3
Pengfei Wang, Zhaoxiang Zhang, Zhen Lei, and Lei Zhang. Sharpness-aware gradient matching for domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3769-3778, 2023. 3
Yan Wu, Jiahao Wang, Yan Zhang, Siwei Zhang, Otmar Hilliges, Fisher Yu, and Siyu Tang. Saga: Stochastic whole-body grasping with contact. In Computer -SCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part VI, pp. 257-274. Springer, 2022. 2, 3, 25, 33
Xianghui Xie, Bharat Lal Bhatnagar, and Gerard Pons-Moll. Chore: Contact, human and object reconstruction from a single rgb image. In Computer Vision-ECCV 2022: 17th European Con- ference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part II, pp. 125-145. Springer, 2022.
Lixin Yang, Xinyu Zhan, Kailin Li. Wenqiang Xu, Jiefeng Li, and Cewu Lu. Cpl: Learning contact potential field to model the hand-object interaction. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 11097-11106 2021. 33
Yufei Ye, Poorvi Hebbar, Abhinav Gupta, and Shubham Tulsiani. Diffusion-guicde reconstruction of everyday hand-object interaction clips. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 19717-19728. 2023.
He Zhang, Yuting Ye, Takaaki Shiratori, and Taku Komura. Manipnet: neural manipulation synthe- sis with hand-object spatial representation. ACM Transactions on Graphics (ToG), 40(4):1-14, 2021. 2, 3, 5, 15
Ruipeng Zhang, Qinwei Xu, Jiangchao Yao, Ya Zhang, Qi Tian, and Yanfeng Wang. Federated do- main generalization with generalization adjustment. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3954-3963, 2023. 3
Yan Zhang and Siyu Tang. The wanderings of odysseus in 3d scenes. In Proceedings of the IEEE/VV Conference on Computer Vision and Pattern Recognition, pp. 20481-20491, 2022. 3
Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang. Domain generalization with mixstyle. In ICLR, 2021a. 7, 32
Kanglei Zhou, Zhiyuan Cheng, Hubert PH Shum, Frederick WB Li. and Xiaohui Liang. Stgae: Spatial-tempora graph auto-encoder for hand motion denoising. In 2021 IEEE International Symposium Mixed and Augmented Reality (ISMAR), pp. 41-49. IEFF, 2021b. 2,3, 5, 15
12

---

Published as a conference paper at ICLR 2024
Keyang Zhou, Bharat Lal Bhatnagar, Jan Eric Lenssen, and Gerard Pons-Moll Toch: Spatio- temporal object correspondence to hand for motion refinement. In European Conference on Computer Vision (ECCV). Springer, October 2022. 2, 5, 6, 7, 15, 30, 31, 32, 33
Qian-Yi Yi Zhou, Jaesik Park, and Vladlen Koltun. Open3D: modern library for 3D data processing. arXiv:1801.09847, 2018. 31
13

---

Published as a conference paper at ICLR 2024
Overview. The Appendix provides list of materials to support the main paper.
Additional Technical Explanations (Sec. A). We give additional explanations comple- ment the main paper.
The GeneOH representation (Sec. A.I). We discuss more insights into the representa- tion design, why GeneOH can highlight errors, and how it compares to representations designed in previous works.
GeneOH Diffission (Sec. A.2). We talk more about the whitened noise space, the dif- fusion process, the step stochastic denoising process, the "denoising via dif- fusion" strategy, and the progressive denoising, together with a discussion on why each denoising stage can successfully clean the imput without breaking the naturalness achieved after previous stages.
Fitting for a hand mesh trajectory (Sec. A.3). We provide details of the fitting process.
Additional Experimental Results (Sec. B). We include more experimental results in this section to support the effectiveness of the method, including
HOI denoising results (Sec. B.l). We include more denoising results GRAB, GRAB (Beta), HOI4D, and the ARCTIC dataset, including long sequences with bi- manual manipulations. Besides,
discuss the results of another series of experi- ments where the training dataset is changed to the training set of the ARCTIC dataset.
Ablation studies (Sec. B.2). We provide more quantitative and qualitative results of the ablation studies.
Applications (Sec. B.3). We provide more results the applications that model can support.
Failure cases (Sec. B.4). We discuss the limitations and failure cases of our method.
Analyzing the distinction between noise in real hand-object interaction trajectories and artificial noise (Sec. B.5). We discuss the differences between the real noise pat- terns and the artificial noise.
User study (Sec. B.6). We additionally include a user study to further assess the quality of our denoised results.
Experimental Details (Sec. C). We illustrate details of datasets, metrics, baselines, mod- els, the training and evaluation settings, and the running time as well as the complexity analysis.
We include a video and an website to introduce our work. The website and the video contain animated denoised results. We highly recommend exploring these resources for an intuitive under- standing of the challenges, the effectiveness of our model, and its superiority over prior approaches.
A ADDITIONAL TECHNICAL EXPLANATIONS
A.1 THE GENEOH REPRESENTATION
More insights of the canonicalization design on T are explained as follows.
Canonicalization design on the temporal relations T. The temporal relations T leverages hand- object relative velocity v at cach frame & of each hand-object point pair (h, o) to represent the motion relations. We further canonicalize the relative velocity via object normals by decomposing vho into two statistics: vho vertical to the object tangent plane and parallel to the object point normal, and hoo E lying in the object's tangent plane. This decomposition enables the model to learn different denoising strategies for the two types of relative velocities and enhance cross-interaction generalization by factoring out object poses. However, relying solely on relative velocities is in- sufficient to reveal motion noise hand-object interactions. The same relative velocity parallel to the normal direction noisy state when they are contact. To address this, we further encode the distance between each hand-object pair and their relative velocities into two statisties, (( k.]] - )), using the following
correspond to clean state when the hand is far from the object, but
14

---

Published as a conference paper at ICLR 2024
formulation:
kk.1 6
ho
k-uh
kvll bollvv
R
ho ex.]
- ka | v..l
(3)
Here, k; ka, and hi are positive hyperparameters, and the term k is inversely related to the distance between the hand and object points. This formula allows the statistics for very close hand- object point pairs to be emphasized in the representation.
Why GeneOH can highlight errors? For spatial errors, we mainly consider the geometric pen- etrations between the hand and the object. The hand-object spatial relations in GeneOH reveal penetrations by parameterizing the object normals and hand-object relative offsets. In more detail, for each hand-object point pair (hg- Og), the dot product between the object normal n, of og and the relative offset hςg o₁ indicates the signed distance between the object point and the hand point. For each hand point h its signed distance to the object mesh can be revealed by jointly considering its signed distance to all generalized contact points. Since the hand point h& penetrates the object if and only if its signed distance to the object mesh is negative, the spatial relation parameterization S can indicate the penetration phenomena.
For temporal errors, we mainly consider inconsistent hand-object motions. There is no unified def- inition statement regarding what consistent hand-object motions indicate. Intuitively, the hand should be able to manipulate the object, where sufficient contact and consistent motions between very close hand-object point pairs are demanded. For very close hand-object pairs, the sliding mo- tion on the object surface is permitted but the vertical penetration moving tendency is not allowed. The above expectations and unnatural situations can be revealed from simple statistics like hand- object relative velocities and hand-object distances. The distance can tell whether they are close to each other. The relative velocity can reveal their moving discrepancy. The decomposed rela- tive velocity lying in the tangent plane and vertical to the tangent plane indicate the surface sliding tendency and the penetrating tendency respectively. The cettancerrelttdd weight term e < can emphasize hand-object pairs that are very close to each other. Therefore, the temporal relations rep- resentation 'T leveraged in GeneOH can successfully indicate the temporal naturalness and incorrect phenomena. Thus learning the distribution of TT can teach the model what is temporal naturalness and how clean the noisy representation.
The GeneOH representation can be applied to parameterize interaction sequences involving rigid or articulated objects. It carefully integrates both the hand motions and hand-object spatial and tem- poral relations more expressive and comprehensive compared to designs in previous works (Zhou al., 2022: 2021b; Zhang et al., 2021). Besides, it can highlight spatial and temporal interaction errors. The above advantages make GeneOH well-suited for the HOI denoising task.
Inspecting back to previous works, TOCH (Zhou et al., 2022) does not explicitly encode hand- centric poses or hand-object temporal relations and only grounds the hands onto the object without careful consideration object distances to represent their relations. But this is not enough to reveal their spatial relations. Canonicalizations are also not carefully considered in this work.
cross-interaction alignment. ManipNet (Zhang et al., 2021) takes hand-
A.2 GENEOH DIFFUSION
We give more detailed explanation of the three denoising stages in the following text.
The whitened noise space. This space is constructed by diffusing the training data towards a random Gaussian noise. A diffusion timestep < lain < T where T is the maximum timestep controls to what extent the input is diffused to pure Gaussian noise. It is the space modeled by the diffusion models during training. The diffusion function we adopt in this work is also exactly the same as the forward diffusion process of diffusion models. To be more specifically, given a data point ., the fair diffusion would transform to F, by linear combination of x and a random Gaussian noise n with the same size to x via the following equation:
an where a, = 1 B, = a, {3. is the forward process variances. The distribution of x, is normal distribution: Ia ((Vo, (1 vo,)
It Vara
15

---

Published as a conference paper at ICLR 2024
Intuitively, the noise space contains all possible 2; across all possible timestep 1 < t < T. 21 with smaller will be more similar to 2. In practice, T is set to 1000, 81 = 0.001, Br = 0.02
with linear interpolation between them to create the variance sequence }. During training t is uniformly sampled from the (4/ E Z, v t t T 7).
Training of the denoising model. Denote the multi-step stochastic denoising model leveraged in our method as denoise(:; t) which takes the noisy sample with the noise scale t as input and denoise 1. When
it back to the noise sample with noise scale clean data manifold depicted by the model and is taken as the final denoised result. The denoise(:, t) leverages a score function eg t)) to predict the noise component of the input noise sample Tt. The score function eg(-,t) contains optimizable network weights 0 and is what we need to learn during training. <o(;;t) 1) only predicts the noise component n. After that, posterior sampling process is leveraged to sample - based on 3t and the predicted ń via the following equation:
1, the denoised result lies in the
1 (8x
1 a'e co(ae, t)) o.Z, v Ckt
11-1
(5)
where z E (O, of 3, Therefore, the denoising model is a multi-step stochastic denoiser since each step it only identifies the mean of the posterior distribution and the denoised result needs to be sampled from the distribution with the predicted mean and the pre-defined variance.
The "denoising via diffusion" strategy. The input trajectory with noise is diffused to ž via Gaussian noise with the diffusion timestep taift using the following equation:
Eun Stan =
Vtaan + v Oltan D,
(6)
where n E (0, I) is a random Gaussian noise.
Given the noisy sample ā. with noise scale I, the denoising model denoise(;, () predicts its noise component via the score function:
= <= 4).
Then Tq is denoised to ãz-1 with noise scale ( 1 by sampling from the posterior distribution following Eq. 5 with the predicted mean ã, 1-a1 á and the pre-defined variance of
vI-a.
DitionDiff: canonicalized hand trajectory denoising. This denoising stage removes noise from the canonicalized hand trajectory jimput of the input noisy interaction sequence by applying the diffusion model for one stage-denoised Jatuge following the "denoising via diffusion" strategy-
To do this, the noisy representation finput is diffused by adding noise for tm steps, followed by denoising for im steps using the diffusion model. The resulting one stage-denoised hand trajectory Jatgg,, in the world coordinate space is obtained by de-canonicalizing the denoised canonicalized hand trajectory Jstage, using the pose of the generalized contact points (Ru; tx). GeneOHinput can S fratue)).
also be updated accordingly into GeneOH (Jauge, Saauge,
SpatialDiff: hand-object spatial denoising. The hand-object spatial denoising module operates on the noisy hand-object spatial relations Sange, of the one stage-denoised interaction sequence output by the previous MotionDiff stage. The representation Sstags, is diffused by adding noise for t, diffusion steps, followed by another t, step of denoising. Once we obtain the denoised representation S which includes the hand-object relative offsets {(h o centered at each generalized contact point oy, we adopt a simple approach to convert it into a two dennisd hand sequence. Specifically, we average the denoised hand offsets from each object point as follows:
J = Average (ha - ox) + O Ok E Pk).
(8)
Following this, GeneOH will be updated to GeneOH {Fstage, S j
TemporalDiff: hand-object temporal denoising. We proceed to clean the noisy hand-object tem- poral relations 7 of the two stages-denoised sequence. The "denoising via diffusion" procedure applied to the temporal relations to achieve this. We then add additional optimization dis- till the information contained in the denoised temporal representation to the three stages-denoised trajectory. The objective is formulated as:
minimizel| f(3 (().P))+7 Jstage, P) - Taagee Juurz
(9)
16

---

Published as a conference paper at ICLR 2024
where P is the sequence of generalized contact points, (i...))--7(.)) converts the hand trajectory to the corresponding temporal relations. The distance is calculated on hand-object distances, i.e, {dho), ), relative velocity {vke) and two relative velocity-related statistics ( {( ch ). We em- ploy an Adam optimizer to find the optimal hand trajectory. The optimized trajectory taken as the final denoised trajectory J.
Stage-wise denoising strategy. Let I where M is the manifold contains all interaction sequences. Let M,g Mss, and Mr represent the manifolds depicted by the three denoising stages respectively. Let Iz. Is. and IT represent one stage-denoised trajectory, two stages-denoised trajectory, and the three stages-denoised trajectory respectively. Further, let Rg. R.s. and R2 denote the set of all natural canonicalized hand trajec- tories, natural hand-object spatial relations, and correct hand-object temporal relations respectively. Let R. Rs,, and Ry denote the set of all canonicalized hand trajectories, hand-object spatial re- lations, and hand-object temporal relations respectively. Denote the function that transforms the interaction trajectory I to the canonicalized hand trajectory as fz-J. the function that converts Z to the hand-object spatial relations as fI-ss, and the function that transforms I to the hand-object temporal relations as fI+T.
((He, Oz) E M denote interaction sequence,
For all interaction trajectories considered in the work, we make the following assumption: Assumption For any trajectory I with the first frame free of spatial noise, we can find natural trajectory I with the same frame, that is I' = I[]].
The three fully- fully-trained denoising models for J. S, and T should be able to map the corresponding input representation to the set of Rg. Rg. and Rg respectively. Then the relations between the interaction manifolds depicted by the three denoising stages and the natural data prior modeled by the three denoising models have the following relations:
. I E .Mg if and only if (Z) E Rg:
I E MMs if and only if fr-+((T) E Rg:
. I E Mr if and only if fz-T (I) E RR and the first frame Σ[1] is free of spatial noise.
Based on the relations between J. and T, we can make the following claim:
Claim There existing functions fs-J Rs - R3 and fr-s (R,, . - Rss,, that for any interaction I with corresponding GeneOH representations (T) (J,8, T) we have: J = Is--((S) and S = [T=
Proof. The canonicalized hand keypoints each frame k, i.e., Ja is composed of each canoni- calized hand keypoint Js = {hy tx tx) R hh E JR). which can be derived from the canon- icalized hand-object spatial relation at the frame k. Specifically, for each Oş E Pk (hg tx RT = (h2- O₂ RT + [Ox- tx RT. The unique canonicalized hand trajectory at the frame k can be decided from the trajectory converted from each object point og E Pk Depending on the conversion function from such multiple hypotheses of the canoniculized hand trajectory re- sulting from different Ok, there exists a function fs-gJ S - R.g that transforms the hand-object spatial relations S the canoncialized hand trajectory J.
have
Similarly, given the hand-object temporal relations at the frame k(1 k of the object point Og Ps and the natural hand keypoints at the starting frame 1, J...,.,,, the relative velocity for each hand-object pair (hg; ok) can be derived from the decoded hand-object relative velocity vho two relocity-related statisfics (cho <h,)). the hand-object distance also Given the hand- object relative positions {(h, 01) h1 E J.J. the hand-object relative positions at each following frame ho + 1(1 < k I K K-11
be derived iteratively via the hand-object relative velocity (voo Ob E P ha+1 0k+1 (h2 0k) + Alv Therefore, there existing a function
T RT + Rs that can convert the temporal relations TT to the hand-object spatial relations S.
Based on this property, we can make the following claim regarding the relations between the three gradually constructed manifolds:
Claim Assume the first frame of the two denossde trajectory [s[1] is free of spatial noise, which almost always holds true, we have MT MMs C MM..
17

---

Published as a conference paper at ICLR 2024
Proof. For I MM with the GeneOH representation (J.S,T} assume I & Mg.
From I E Mss we have S R.::
Based on the definition R's, the set of spatial relations derived from all natural interac- tions, there exists a natural interaction I' so that fI=+s(I')
Since I' a natural interaction, we have J' I'= R::
Since J Is-a((S) J', we have I Rigi
Based on the assumed fully-trained denoising model, we have I E Mgg.
The conclusion contradicts with the assumption I & MM-- Thus Ms S M3 holds true.
For I E .M with the GeneOH representation {J,S, T3 whose first frame is free of spatial noise, assume I # .Ms.
From I E MT, we have that T E Rg:
Based on the definition of RGG, the set of temporal relations derived from all natural inter- actions, and the Assumption there existing a natural interaction I', with the first frame same to I, so that (I'+ =
Since I' is natural interaction, have S =I-ss(I') Rs::
Since S = fT fT- (T,[)) S', have S Rg:
Based the assumed fully-trained denoising model, we have I Ms.
The conclusion contradicts with the assumption I .Ms. Thus Mr S Ms holds true.
The stage-wise GeneOH Diffusion functions the following steps clean the input interaction M:
Given the input interaction I, the denoising model for I maps J to another J Ri-- There existing an interaction Ia s.t. JL = fI=+ (I+), which is also exactly the same as the trajectory derived from J and the object trajectory {Ox) Therefore, after the first denoising stage, we have I MMg.
Given St the denoising model for S maps St S Rg. There existing an interaction I s.t. 8 = fz-s( Therefore, after the second denoising stage, we have I e MMs-
After that, the denoising model for T maps T = fI-T(IT)) to T Rs. After that, the interaction I constructed the following steps:
Construct 5 from 3 and [1 via S3 s (T3 II((1));
Construct J from S via J fs-z($9)
- Construct 3 from ja and the object trajectory {O*)K
Since T fI Rÿ and [ [2 is free spatial noise, we have Z MT.
Therefore, the three denoising stages gradually map the input noisy interaction to progressively smaller manifold contained in the previous large manifold. Formally have
GeneOH Diffusion(-) M + My + Ms + MT.
(10)
A.3 FITTING FOR AN HOI TRAJECTORY
Once the interaction sequence has been denoised, we proceed to fit a sequence of hand MANO (Romero et al., 2022) parameters obtain final hand meshes. The objective is optimiz- ing a series of MANO parameters (Ię. ty. Bg. 0x) so that they fit the denoised trajectory J well. Notice the hand trajectory J consists of sequence of hand keypoints
also need to derive
18

---

Published as a conference paper at ICLR 2024
Table 3:
HOI4D Dataset. Per-category statistics the HOI4D dataset used in our experiments, including number of sequences and the index of the start frame.
Laptop
Pliers
Scissors
Bottle
Bowl
Chair
Mug
ToyCar
Kettle
#Seq. Starting Frame
155
187
93
214
217
167
249
257
58
120
150
50
0
0
0
0
0
0
keypoints from MANO parameters to allow the above optimization. Luckily this process is differen- tiable and we use Jrecon [rr, tjp, Bk. 0g ] k==)) to denote We can therefore optimize the following reconstruction loss for the MANO hands
Lrecon = || Jreeco URs 8k, On )
(11)
where the distance function is simple mean squared enror (MSE) between the hand keypoints at each frame. To regularize the hand parameters {.kk 0k} and enforee temporal smoothness, we introduce an additional regularization loss defined as
K
Creg
Σ
0k+1 0x
(12)
K
k=1
The overall optimization objective is formalized
minimize (ra.tiieeerrrrr.. (e..tu,.6,001) (Crecon + Lreg);
(13)
and we employ an Adam optimizer to solve the fitting problem.
B ADDITIONAL EXPERIMENTAL RESULTS
We present additional experimental results to further support the denoising effectiveness and the strong generalization ability of our method.
B. HOI DENOISING
For the first set of evaluation metrics, Table presents more evaluations on our method, ablated versions, and the comparisons to the baseline models than the table in the main text. For the sec- ond set of evaluation metrics, 'able 5 summarizes more results and comparisons. For stochastic denoising methods, including "Ours", "Ours w/o Canon.", "Ours w/o SpatialDDiff and "Ours w/o TemporalDiff", we report the mean and the standard deviation of results obtained from three inde- pendent runs with the random seed set to 11, 22, and 77 respectively. Such statistics offer a more comprehensive view of the average results quality produced by those methods. Please notice that the evaluation method is different from the one present in the main text, where the result closest to the input trajectory among 100 independent runs is chosen to report evaluation metrics.
More results on the GRAB test novel interactions with new objects. Figure shows quali- tative evaluations on the GRAB test set to compare the generalization ability of different denoising models towards novel interactions with unseen objects.
More results on the GRAB (Beta) test set novel interactions with new objects and unseen synthetic noise patterns. Figure shows qualitative evaluations on GRAB (Beta) test set to compare the generalization ability of different denoising models towards unseen objects, unobserved interactions, and novel synthetic noise.
More results on the HO14D dataset novel interactions with new objects and unseen real noise patterns. Figure 10 shows qualitative evaluations on the HOI4D test set to compare the generalization ability of different denoising models towards unseen objects, unobserved interactions, and novel real noise.
Wired hand trajectory produced by TOCH (w/ MixStyle). Through experiments with TOCH on interaction sequences with new noise patterns unseen during training, we frequently observe strange hand trajectories from its results with stiff hand poses, unsmooth trajectories, and large
19

---

Published as a conference paper at ICLR 2024
Table 4: Quantitative evaluations and comparisons. Performance comparisons of our method,
baselines, and ablated versions on different test sets using the first set of evaluation metrics. Bold
red numbers for best values and italic blue values for the second hest-performed ones.
MPJPE (mm, 4)
MPVPE (mm, L)
C-IoU
Dataset
Method
(%, f)
Input
23.16
22.78
1.01
TOCH
12.38
12.14
23.31
TOCH (w/ MixStyle)
13.36
13.03
23.70
GRAB
TOCH (w/ Aug.)
12.23
11.89
22.71
Ours (w/o SpatialDiff)
7.83
7.67
26.09
Ours (w/o TemporalDiff)
8.27
8.13
26.55
Ours (w/o Diffusion)
8.52
8.38
26.44
Ours (w/o Canon.)
10.15
10.07
24.92
Ours
9.28
9.22
25.27
Input
17.65
17.40
13.21
TOCH
24.10
22.90
16.32
GRAB (Beta)
TOCH (w/ MixStyle)
22.79
21.19
16.28
TOCH (w/ Aug.)
11.65
10.47
24.81
Ours (w/o Diffusion)
12.16
11.75
22.96
Ours (w/o Canon
10.89
10.61
24.68
Ours
9.09
8.98
26.76
Input
25.51
24.84
1.68
ARCTIC
TOCH
14.34
14.07
20.32
TOCH (w/ MixStyle)
13.82
13.58
21.70
TOCH (w/ Aug.)
14.18
13.90
20.10
Ours
11.57
11.09
23.49
penetrations shown in Figure 9 and 4. Such phenomena cannot be mitigated by augmenting it with general domain generalization techniques. Figure 12 demonstrates that the improved version, TOCH with MixStyle, also yields similar unnatural results. This suggests that the novel noise distri- bution presents a challenging obstacle for the denoising model to generalize to new noisy interaction sequences. In contrast, method does not have such difficulty in handling the shifted noise distri- bution.
Results of TOCH and TOCH (w/ Aug.) the HO14D dataset. Figure 11 compares the results of TOCH (w/ Aug.) with our method. In the example of opening a scissor, TOCH produces very strange "flying hands" trajectories, for which please refer our video for an intuitive understanding. Though the results produced by the improved version do not exhibit the "flying hands" artifacts, it still very strange, stiff, suffering from very unnatural hand poses, and cannot perform correct manipulations. The results of TOCH and TOCH (w/ Aug.) on the ToyCar example are very similar since our experiments indeed get very similar results from such two models in this case. They both are troubled by strange hand shapes and very unnatural trajectories.
More results on the ARCTIC dataset novel interactions with new objects involving dynamic object motions and changing contacts. Figure 13 shows qualitative evaluations on the ARCTIC test set to test the ability of our denoising model to clean noisy and dynamic interactions with changing contacts.
Besides, we include samples of our results on longer sequences with bimanual manipulations in Figure 14.
Multi-state denoising v.s. one-stage denoising. We leverage a multi-stage denoising strategy in this work to tackle the challenge posed by complex interaction noise. In Section A.2, demonstrate the stage-wise denoising strategy gradually projects the input trajectory from the manifold containing unnatural interactions, to the manifold of trajectories with natural hand motions, the manifold
20

---

Published as a conference paper at ICLR 2024
Table 5: Quantitative evaluations and comparisons. Performance comparisons of our method, baselines, and ablated versions on different test sets using the second set of evaluation metrics. Bold red numbers for best values and italic blue values for the second best- rest-performed ones. "GT" stands for Stround-Truth
Penetration Depth
Proximity Error (mm, 4)
Motion Consistency
Dataset
Method
(em²),
(mm, 4)
(mm
GT
0,50
1.33
0.51
Input
4.48
5.25
13.29
881.23
TOCH
2.09
2.17
3.12
20.37
TOCH (w/ MixStyle)
2.28
2.62
3,10
21.29
GRAB
TOCH Aug.)
1.94
2.04
3,16
22.58
Ours (w/o SpatialDiff)
2.15+00.22
2.29++0.03
6.71+1009
12.16+00.67
Ours (w/o TemporalDiff)
0.8622.02
1.5420002
3.9310.31
9.36::0.88
Ours (wlo Diffusion)
1,07
1,70
2.63
10.05
Ours (w/o Canon.)
1.57:10.02
1.3310.03
20,22
1.30:00033
Ours
1,22+00011
1,72+0.01
2.44+0.18
0.41+0001
GT
0.50
1.33
0
0.51
Input
2.19
4.77
5,83
27.58
GRAB (Beta)
TOCH
2.33
2.77
5.60
25.05
TOCH (w/ MixStyle)
2.01
2.63
4.65
17.37
TOCH (w/ Aug.)
1.52
1.86
3,07
13.09
Ours (w/o Diffusion)
1.98
2.06
3.00
11.99
Ours (w/o Canon.)
1.79.10.2
1.73.10.03
3.19.10.15
1.28.10.03
Ours
1.18+0.00
1.69=0.01
2.78+00.14
0.54+00.00
Input
2.26
2.47
46.45
TOCH
4.09
4,46
35,93
TOCH (w/ MixStyle)
4.31
4.96
25.67
HOI4D
TOCH (w/ Aug.)
4.20
4.51
25.85
Ours (w/o Diffusion)
3.16
3.83
18.65
Ours (w/o Canon.)
2.37+00.02
3.57:40.03
12.80+0.779
Ours
1.99++0.02
2.14++0.02
9.75++0.8
GT
0.33
0,92
0
0.41
Input
2.28
15.21
931.69
ARCTIC
TOCH
1.84
2.01
4.31
18.50
TOCH (w/ MixStyle)
1.92
2.13
4.25
18.02
TOCH Aug.)
1.75
1.98
22.57
Ours
1.35 +
1.91 0.02
2.69 0.11
0.85 + 0.00
Table 6: Quantitative evaluations of the model trained on the ARCTIC training set. Bold red numbers for best values. "GT" stands for "Ground- Truth".
MPIPE
MPVPE
C-loU (%.
Penetration Depth (mem,
Proximity Error (mmm,
HO Motion Consistency
Datasel
Method
(nmm.
(ema
(mmm?
GT
0.50
1.33
0,51
GRAB
Input
1.01
5.25
Ours (GRAB)
9.28
9.22
25.27
1.23
1.74
2.53
0.57
(ARCTIC)
24.79
0.55
2.26
2.47
46.45
HOHD
Ours (GRAB)
1.99
2.15
9.81
Ours (ARCTIC)
1.54
1.96
9.33
GT Input
0.33
0.92
0.41
ARCTIC
25.51
24.84
1.68
2.28
4.89
15.21
931.69 16.66
Ours (GRAR)
11.57
11.09
23.49
1.35
1.93
2.71
0.92
Ours (ARCTIC)
10.34
10.07
25.08
1.21
2.62
1.10
with correct spatial relations, and to the manifold of trajectories with consistent temporal relations.
One may question whether is possible to use the last projection step only to project the input to the natural interaction manifold in single step. Our experimental observations show the difficulty of removing such complex noise in one single stage. An effective mapping to clean such complex noise is very hard to learn for neural networks.
21

---

Published as a conference paper at ICLR 2024
View
View
View 3
Irput
Nebay
Oun
TOCH
NoiRy Input
TOCH
Nouty
Figure 8: Evaluation and comparisons on the GRAB test set. Input and denoised results are shown from three views via four keyframes in the time-inereasing order. Please refer to our website and video for animated results.
Generalize from ARCTIC to other datasets. To further evaluate the generalization ability of method, we conduct a new series of experiments where we train the model on the ARCTIC dataset (see the Section C for data splitting and other settings) and evaluate on GRAB, HOI4D, and ARCTIC (test split). Table 6 contains its performance. We can observe that though our model trained on the GRAB can generalize to ARCTIC with good performance, the reduced domain gap when using ARCTIC the training set can really improve the performance. For instance, Figure 15 shows that the model trained on the ARCTIC training set can perform obviously better on examples where the model trained on GRAB would struggle (please see Section B.4 for the discussion on failure case). For the sequence where the hand needs to open wide to hold the microwave, the model trained on GRAB cannot clean the noisy very effectively, producing results with obvious penetrations and the unnatural hand trajectory with instantaneous shaking. However, the model trained on the ARCTIC dataset eliminate such noise and produce much natural trajectory. Besides, training on this
22

---

Published as a conference paper at ICLR 2024
View
View
View
Noiey
Inpor
Oues
TOCH
Noisy
Outs
TOCH
Novey Input
Ours
TOCH
Figure 9: Evaluation and comparisons on the GRAB (Beta) test set. Please refer to our website and video for animated results.
dataset can benefit the model's performance on the HOI4D dataset with articulated objects and articulated motions.
B.2 ABLATION STUDIES
This section includes more ablation study results to complement the selected results in the main text. Table 4 and present a more comprehensive quantitative evaluation of ablated models and the comparisons to the full model.
Generalized contact-centrie parameterizations. Apart from the results present in Table and 5, Figure 16 gives and visual example where the ablated version without such contact-centric design cannot generalize well to the manipulation sequence with large object movements. We can still observe obvious penetrations from all three frames present here.
Denoising capability on recovering ground-truth and modeling high-frequency pose details. Together with the ability to model various solutions, the stochastic denoising process also empowers the model to explore a broad space that is more likely to encompass samples close to the ground- truth sequences. Figure 17 shows that ours is more faithful the ground-truth sequence than the result of TOCH, regarding both recovered hand poses and the contact information.
Besides. taking advantage of the power of our HOI representation GeneOH and the novel denoising scheme, we are able to model high-freug high-frequency shape details in the results. However, TOCH's results would exhibit flat hand poses frequently. This may result from its high-dimensional representations
23

---

Published as a conference paper at ICLR 2024
View
Note
TOCH
Nokey
T0CH
Notey
TOCH
Nony
TOCH
Figure 10: Evaluation on the HO14D dataset. Please refer to our website and video for animated results.
24

---

Published as a conference paper at ICLR 2024
View
View 2
View
Noty
Ippor
( AURA
700
TOCTH
Nolyy Inport
TOCH
Figure II: Comparisons on the HO14D dataset. We compare our method with the baseline TOCH and its improved version TOCH (w/ Aug.).
Weird artifacts produced by TOCH (w/ MixStyle). (First line:) Ours result. (Second line:) The result TOCH (w/ MixStyle). The noisy input is perturbed by noise sampled from Beta distribution, different from that used in training.
Figure 12:
and the limited ability of the denoising strategy, which models the deterministic, one-step noise-to- data mapping relation.
B.3 APPLICATIONS
This section presents applications of the denoising model.
Refining noisy grasps produced by the generation network. In addition to refining noisy interac- tion sequences, our method can serve effective post-processing tool to refine implausible static grasps produced by the generation network as shown in Figure 6. Examples shown here are grasps taken from interaction results produced by (Wu al., 2022).
25

---

Published as a conference paper at ICLR 2024
laput
Our
Inpe
Oun
Inpat
topet
Naiey
Input
Timestamp
Evaluation on the ARCTIC dataset. The model cleans noisy right hand trajectory here. Left hands shown both the noisy input and the denoised trajectory are GT shapes. Please refer to our website and video for animated results.
Figure 13:
Noisy
Input
Denoised
Figure 14: Evaluation on long interaction sequences with bimanual manipulations. The model cleans both the noisy right hand trajectory and the noisy left hand trajectory here. Please refer to our website and video for animated results.
26

---

Published as a conference paper at ICLR 2024
Trained GRAB
Trained
ARCTIC
Hold mierowave
Comparisons between the model trained on ARCTIC and the one trained on GRAB. The model trained ARCTIC training set generalize to the corresponding test quences more easily thanks to the reduced domain gap.
Figure 15:
Figure 16: Effectiveness of the generalized contact-centric parameterization. (First line:) Re- sults Ours (w/o Canon.). (Second line:) Results of our full model.
Apart from the denoising ability, the denoised data with high-quality interaction sequences and static grasps can further aid a variety of downstream tasks. Here we take the grasp synthesis and the manipulation synthesis task as example.
Grasp synthesis.. We select four objects and their corresponding grasping poses from the GRAB test set to train the synthesis network. Then The results shown in Figure 18 are natural and contact-aware. In contrast, the generated grasps are plausible as shown the leftmost part of Figure 18.
the network to generate grasps for unseen objects.
Manipulation synthesis. We further examine the quality of the denoised interaction data via the manipulation synthesis task. Based the representations and the network architecture proposed in recent manipulation synthesis work', we train manipulation synthesis network using our de- noised data. The network then takes new object sequence as input to generate the corresponding manipulation sequence. As shown in Figure 19, the quality of our data is well suited for learning- based synthesis model. can generate diverse, high-quality manipulation sequences for an unseen object trajectory.
The above two applications indicate the potential value of our denoising model in aiding high-quality interaction dataset creation.
B.4 FAILURE CASES
Figure 20 summarizes the failure cases. Our method may sometimes be unable perform very well in the following situations: 1) When the hand needs to open wide to hold the object, the canoni- calized hand trajectories and the hand-object spatial relations canonicalized around the interaction region may be extremely novel to the denoising model. The model then cannot fully clean penetra- tions from the observations. 2) When the noisy input contains very strange hand motions such as the sudden detachment and grasping presented in Figure 20, the model can remove such artifact but still cannot clean the trajectory perfectly, leaving us remaining penetrations shown in the denoised result. 3) When the hand is opening unseen object with extremely thin geometry, we may still observe subtle penetrations from the results.
27

---

Published as a conference paper at ICLR 2024
Goans-Traa
TOCH
Dunckanpk
Damplec
Figure 17: Comparisons between our model and TOCH on the ability to recover ground-truth interactions. We can explore wide space that encompasses the sample with hand poses (high- lighted in yellow circles) and contacts (in green circles) close to the ground truth. We can model high-frequency poses. However, TOCH's result contains plain poses and cannot recover bending fingers exhibited in the ground-truth shape.
Recalt Nenoock Trathct Nonty Data
Canex
Lepop
Figure 18: Grasp synthesis. Synthesized grasps for unseen objects.
B.5 ANALYZING THE DISTINCTION BETWEEN NOISE IN REAL HAND-OBJECT INTERACTION TRAJECTORIES AND ARTIFICIAL NOISE
From the visualization and animated results shown on the website and the video, the distinctions between the noise exhibited in real noisy hand-object interaction trajectories (e.g., hand trajectories from the noisy HOI4D dataset and hand trajectories estimated from interaction videos) and the artificial noise could be summarized as follows:
The trajectories in HO14D) always present unnatural hand poses, jittering motions, missing contacts, and large penetrations;
Retargeed hand motions always suffer from large penetrations;
The hand trajectories estimated from HOI videos are usually with penetrations and missing contacts;
A common feature that real noisy hand trajectories always present time-consistent arti- facts. However, noisy trajectories with artificial noise added independently onto each frame usually present vime-varying penetrations and unnatural poses.
Besides, as summarized Table 1, the differences between different kinds of noise patterns can be revealed by comparing various metrics calculated on their input noisy trajectories. including metrics to reveal penetrations (IV, penetration depth), hand-object proximity (C-IoU, Proximity Error), and motion consistency.
Further analysis. We further visualize the difference (Ô - @ot) between noisy hand mano pose parameters (0) and the GT values (09t) obtained from the trajectories estimated from videos (the noisy input of the application on cleaning hand trajectory estimations, Sec. 4.4), the difference between the mano pose parameters with artificial Gaussian noise (θ") and the GT values, and the differences between the parameters with artificial noise drawn from the Beta distribution (ūb). By projecting them into the 2-dimensional plane using the PCA algorithm (implemented in the scikit- lear package), visualize their positions from 256 examples in Figure 21. As we see, the real noise pattern very different from artificial noise. this case, the noise of the hand trajectory estimated from videos further exhibits instance-specifi patterns.
28

---

Published as a conference paper at ICLR 2024
Sample
Sample
Sumple
Timestamp
Figure 19: Manipulation synthesis. Synthesized manipulation sequences for the unseen laptop object. Frames shown here from left to right are in time-increasing order.
Denoised
Open wide to hold the unseen large objecr (ARCTIC)
Noisy
Input
Denoised
Extremely unnatural and strange hand-object temporal relations (HO14D)
Denoised
Unseen object with very thin geometry (ARCTIC)
Figure 20: Failure cases caused by the unseen and large object, very strange hand-object temporal relations, and unseen object with extremely thin geometry.
B.6 USER STUDY
To better access and compare the quality of our denoised results to those of the baseline model, we conducted toy user study. We set up website containing our denoised results and TOCH's results on 18 noisy trajectories in a randomly permutated order. Twenty people who are not familiar with the task even have no background in CS are asked rate each clip a score from 1 to 5, indicat- ing their preferences. Specifically, "1" indicates significant difference between the hand motion demonstrated in the video and the human behavior, with obvious physical unrealistic phenomena such penetrations and motion inconsistency; "3" represents the demonstrated motion is plausible and similar to the human behavior, but still suffer from physical artifacts; "5" means a high-quality motion which is plausible with flaws and is human-like. "2" means the quality is better than "1" but worse than "3". Similarly, "4" means the result is better than "3" but worse than "5".
For each clip, calculate the average score achieved by method and TOCH. The average and medium scores across all clips are summarized Table 7. Ours much better than the baseline model.
29

---

Published as a conference paper at ICLR 2024
Red Erinatass Nemaltk 0.5)
Boaak, 20 *9
-00
Figure 21: Visualization on the differences between the mano pose parameters of hand trajectories estimated from videos and the GT values, the difference between the mano pose parameters with artificial Gaussian noise (Qm) and the GT values, and the differences between the parameters with artificial noise drawn from the Beta distribution (0h). The analysis is conducted on 256 trajectories.
For each trajectory, the difference vectors across all frames are concatenated together and flattened single vector.
Table 7:
User study.
GeneOH Diffusion
TOCH
Average Score
3.96
1.98
Medium Score
4.00
1.55
C EXPERIMENTAL DETAILS
C.I DATASETS
GRAB training set. This the training set used all experiments presented the main text. We follow the cross-object splitting strategy used in (Zhou al., 2022) to split the GRAB (Taheri al., 2020) datasct. The training split, containing 1308 manipulation sequences, is used to construct the training dataset. We also filter out frames where the hand wrist is more than 15 away from the object. For each training sequence, we slice it into clips with 60 frames to construct the training set. Sequences with length of less than 60 are not included for training or testing. For models where noisy sequences are required during training, we create the noisy sequence from the clean sequence by adding Gaussian noise the MANO parameters. Specifically, the Gaussian noise is added the hand MANO translation, rotation, and pose parameters, with standard deviations of 0.01, 0.1, and 0.5, respectively.
ARCTIC training dataset. It is the training set of all models in the experiments where we wish to generalize the model trained on ARCTIC to other datasets. Based on the publicly available sequences from the subject with the index "s01", "s02", "s04", "s05", "s06", "s07", "s08", "s09", "s10", we take the manipulation sequences from "s01" for evaluation and those of other subjects for training. For each sequence, we slice it into small clips with window size equal to 60 and the step size set to 60. We filter out clips where the maximum distance from the wrist to the nearest object point is larger than 15cm. The number of all training clips is 2524. Only the right hand trajectory is used for training.
The following text contains GRAB, GRAB (Beta), HO14D, and ARCTIC.
details about the four distinct test sets for evaluation, namely
GRAB (Taheri et al., 2020). The test split of the GRAB dataset, containing 246 manipulation quences, is used to construct the test set. For each test sequence, we slice it into clips with 60 frames using the step size 30. For each test sequence, the noisy sequence is also created by adding Gaus-
30

---

Published as a conference paper at ICLR 2024
sian noise the MANO parameters. The Gaussian noise is added to the hand MANO translation, rotation, and pose parameters, with standard deviations of 0.01, 0.1, and 0.5, respectively.
GRAB (Beta). The GRAB (Beta) test set is constructed from manipulation sequenees from the GRAB test split. For each sequence, the noisy sequence is created by adding noise from the Beta randomly sample noise from the
distribution (8(8, 2)) to the MANO parameters. Specifically, Beta distribution (B(8,2) 2)). Then the sampled noise was by 0.01, 0.05, 0.3 to get noise vectors added to the translation, rotation, and hand pose parameters respectively.
HOI4D (Liu et al., 2022). For the HOI4D dataset, we select interaction sequences with humans ma- nipulating objects from 3 articulated categories, including Laptop, Scissors, and Pliers, and 6 rigid datasets, namely Chair, Bottle, Bowl, Kettle, Mug, and ToyCar, for test. The number of instances included in each category is detailed in Table 3. For each sequence, we specify the starting frame and take the clip with the length of 60 frames starting from it as the test clip. The starting frame set for each category listed in Table 3.
ARCTIC (Fan et al., 2023). The ARCTIC test set for evaluation takes the right hand only since we observe that dexterous manipulation such as articulated manipulations is always conducted by the right hand. For instance, as shown in the example in Figure 4, the left hand holds the capsule machine with no contact change during the manipulation while the right hand first touches the lid, then touches the base, and then opens and close the lid. However, we can refine the left hand motions as well, as demonstrated in the second sequence of the "refining estimation from video" example shown in Figure 1. The manipulation sequences from "s01", 34 in total, are taken for evaluation. For each test sequence, the quantitative results are evaluated from clips with the window size 60 sliced from each sequence using the step size 30. The filtering strategy similar to that used for constructing the training dataset is applied to the test clips as well. The default length of the clips used in the qualitative evaluation is 90, which is the composed result of two adjacent clips with a window size of 60. The noisy sequence is obtained by adding Gaussian noise to the right hand MANO parameters. Specifically, the Gaussian noise is added to the hand MANO translation, rotation, and pose parameters, with standard deviations of 0.01, 0.05, and 0.3, respectively.
Since the ARCTIC's object template meshes do not provide vertex normals, which are demanded both in our method and some baseline models, we use the "compute, cemputee_eettex..mm normals" function im- plemented in Open3D (Zhou al., 2018) for computing the vertex normals.
C.2 METRICS
We include two sets of evaluation metries. The first set follows the evaluation protocol of previ- ous works (Zhou et al., 2022) and focuses on assessing the model's capability to recover the GT trajectories from noisy observations, as detailed in the following.
Mean Per-Joint Position Error (MPJPE). It calculates the average Euclidean distance between the denoised 3D hand joints and the corresponding ground-truth joints.
Mean Vertex Position Error (MPVPE). It measures the average Euclidean distance between the denoised 3D hand vertices and the corresponding ground-truth vertices.
Contact ToU (C-IoU). This metric assesses the similarity between the refined contact map and the ground-truth contact map. The binary contact maps are obtained by thresholding the correspon- dence distance within #2mm. For our method, which does not rely on correspondences introduced in (Zhou et al., 2022), we utilize the computing process provided by (Zhou et al., 2022) to compute the correspondences.
To measure whether the denoised trajectory exhibits natural hand-object spatial relations and con- sistent hand-object motions, introduce the second set of evaluation metrics.
Solid Intersection Volume (IV). We evaluate this metric following (Zhou et al., 2022). It quantifies hand-object piter-penetrations By voxelizing the hand mesh and the object mesh calculate the volume of their intersected region as the intersection volume.
Per- Vertex Maximum Penetration Depth (Penetration Depth). For each frame. calculate the maximum penetration depth of each hand vertex into the object. We then average these values across all frames to obtain the ver-vertex maximum penetration depth.
31

---

Published as a conference paper at ICLR 2024
Proximity Error. The metric is only evaluated on datasets with ground-truth references, including GRAB, GRAB (Beta), and ARCTIC. For each vertex of the denoised hand mesh, we compute the difference between its minimum distance to the object points and the corresponding ground-truth vertex's 's minimum distance to the object points. The proximity error is obtained by averaging these differences over all vertices. The overall metric is obtained by averaging the per-frame metric over all frames. Specifically, let Mmmin denote the minimum distance from the hand keypoint he at the frame k to objects points. Formally, it is defined kmmnn min t = |lh ox||2/he
Jk, Og E P&.. Let di mmin represents the quantity of the keypoint h from the denoised trajectory, and dh kimin represents the quantity of the keypoint h from the ground-truth trajectory. Then the overall
metric calculated as Proximity error = mean ( {Illd, R,min 1imminll2//bn E J=H1 S k S K}.
Hand-Object Motion Consistency (HO Motion Consistency). This metric assesses the consis- tency between the hand and object motions. For each frame where the object is not static, we identify the nearest hand-object point pair (hg, o₂) = argmin (amiin (h2 E J&. Og E Px)]. We use the expression ||e- [|e-100jhbe (o00)b---=]] Ah, - Aox to quantify the level of inconsistency between the hand and object motions. Here, Ahş and Δoş represent the displacements of the hand point and the object point between adjacent frames, respectively. We obtain the overall metric by averaging the metric over all frames.
C.3 BASELINES
We give a more detailed explanation of the compared baselines as follows to complement the brief introduction in the main text.
TOCH. We compare our model with the prior art on the HOI denoising problem, TOCH (Zhou al., 2022). TOCH utilizes an autoencoder structure and learns to map noisy trajectories their corresponding clean trajectories. By projecting imput noisy trajectories onto the clean data manifold, it can accomplish the denoising task. We utilize the official code provided by the authors for training and evaluation.
TOCH (w/ MixStyle). Further, to improve TOCH's generalization ability towards new interactions, we augment it with general domain generalization method MixStyle (Zhou et al., 2021a), resulting variant named "TOCH (w/ MixStyle)".
TOCH (w/ Aug.). Another variant, "TOCH (w/ Aug.)", where TOCH is trained on the training sets of the GRAB and GRAB (Beta) datasets, further introduced to enhance its robustness towards unseen noise patterns.
C.4 MODELS
Denoising models used our method. We realize the denoising model's function and the training as those of the score functions in diffusion-based generative models. We adapt the implementation of Human Motion Diffusion (Tevet et al., 2022) to implement our three denoising models for each part of the representation". Instead of training the denoising function to predict the start data point Xo from the noisy data x, as implemented in (Tevet et al., 2022), we predict the noise (x, - Xo). We also follow its default training protocol.
We mainly adopt MLPs and Transformers as the basic backbones of the denoising model. The detailed structure depends on the type of the corresponding statisties and the dimensions. The code in the Supplementary Materials provides all those details. So we spare the effort to list them in detail here.
When leveraging the denoising model to clean the input via the "denoising via diffusion" strategy, the diffusion steps is set 400 for MitionDiff, 200 for SpatialDiff, and 100 for TemporalDiff empirically.
Ours (w/o Diffusion). In this ablated version, we design a denoising autoencoder for cleaning the spatial and temporal representations. For each representation J, S.S.T. we leverage an autoen- coder for denoising. After that, we get the final hand meshes by fitting the MANO parameters
32

---

Published as a conference paper at ICLR 2024
(rk: tk. Sx,0x) to reconstruct the denoised representations. Assuming the reconstructed hand tra- jectory as Jreeon, the reconstructed hand-object spatial relative positions as Sweeon and the tempo- ral representations Treson, the reconstruction loss is formulated as follows:
Crep recon
- |l2 + > - Srecon + As||T - -T
(14)
where A1, Ag. A3 are coefficients for the reconstruction losses. We set A1, A2. A3 1. our ex- periments. The distance function between the spatial representations is calculated on the relative positions between each point pair. The distance between the temporal representations calculated on hand-object distances, ie,, {d and two relative velocity-related statistics ( ((&,]] ch. Together with the regularization loss
K
Creg E + 110112)) + 1=
0 0x112.
(15)
K
A=1
the total optimization target is formulated as follows,
(rx.tx..e.99)) minimize minimize(ra..t.... A (Frecon + Ereg);
(16)
and we employ an Adam optimizer to solve the problem.
TOCH (w/ MixStyle). We use the official code provided to implement the MixStyle layer. We add a MixStyle layer between every two encoder layers of the TOCH model (Zhou et al., 2022).
Configurations of MixStyle are kept the same as the default setting.
TOCH (w/ Aug.). The model is trained on paired noisy-clean data pair from the GRAB training set. We perturb each training sequence with types of noise, that is the noise from Gaussian distribution, and the noise from a Beta B(8, 2) distribution. The noise scale for the Gaussian for the translation, rotation, and hand poses are 0.01, 0.1, and 0.5 respectively. The scale of the Beta noise added on the translation, rotation, and hand poses are 0.01, 0.05, and 0.3.
Grasp synthesis network. We adapt the WholeGrasp-VAL VAE network proposed in (Wu et al., 2022) to a HandGrasp-Ve VAE network?. Instead of using whole-body markers, we use hand anchor points (Yang et al., 2021), composed of 32 points from the hand palm in total. To identify contact maps for both hand anchors and object points, we set distance threshold, i.e., 2 mm, and mark the status of points with the minimum distance the hand/ohject as contact. During training, we do not add the ground contact loss since the whole body is not considered in our hand-grasping setting. To train the network, we further split the GRAB test set into a subset containing binoculars, wineglass, fryingpan, and mug for training. Then we use the network to synthesize grasps for unseen objects. We select 100 grasps for each object to construct the training dataset.
Manipulation synthesis network. We utilize the denoised manipulation trajectories for the Laptop category to train the manipulation synthesis network. The training data consists of 100 manipulation sequences.
C.5 TRAINING AND EVALUATION
The denoising model for J. The denoising model for the canonicalized hand trajectory J is trained on canonicalized hand trajectories (T} of all interaction sequences the training set. We apply per-instance normalization operation to those points at each frame for centralization and scaling purposes. Specifically, we utilize the mean and the standard deviation statistics calculated for all points across all frames. In more detail, we first concatenate all keypoints over all frames to form the concatenated keypoints J Jeoncat,
Connaa Concat (Ja, dim 0)
(17)
where Ju E RR for each frame k. Then, the average and the standard deviation is calculated on Jooncat via
1 Average conncat, dim 0)
(18)
Std Joncatt dim = 0).
(19)
33

---

Published as a conference paper at ICLR 2024
The (1e, o0)) are utilized to normalize J& at each frame k. i.e.
Jg JR
(20)
σ
The denoising model for S. Similarly, the denoising model for hand-object spatial relations S trained using representations {S} from all interaction sequences in the training set. We apply per ver-instance normalization to the canonicalized hand-object relative positions {(h, o ) ). The normalization conducted in a per-instance per-object point way. For each object point o in generalized contact points P, we calculate the average and standard deviation of {{hs ou } over all frames k. We first concatenate the relative positions over all frames and all hand keypoints for the concatenated spatial relations, denoted as
concat Concat (he - ) dim 0] Then, the average and the standard deviation is calculated on Concat via
(21)
Jea = Average(S dim = 0)
(22)
a² (S dim 0).
(23)
Such statistics (10,0% utilized to normalize the relative positions {(h; -ok),ie. i.e.
(he 02) (h+ ok) n
(24)
The denoising model for T. When training the denoising model for the hand-object temporal relations T. we first train an autoencoder, composed of an encode(-) function and a decode(-) function for T. It takes the T as input and decode the hand-object distances {dhho } and the relative velocity-related quantities {eh. th Then, the denoising model is trained the en- coded latent (T)code(T)). This approach avoids the need for designing normalization strate- gies for the temporal representations. We adopt a PointNet structure block with positional en- coder followed by transformer encoder module for encoding the temporal relation representa- tions. Given the input temporal representation T E RKXNN the PointNet encoder block passes it through four PointNet blocks each with three encoding layers with latent dimension (32, 32, 32), (64, 64, 64), (128, 128, 128), (256, 256, 256) respectively. The transformer encoder feedforward latent dimension 1024, dropout rate O,
module is with parameters "num heads" and the latent dimension 256. The decoder contains fully connected layers for decoding each kind of statistics dho L.k l|.b individually.
Train-time rotation augmentation. For the canonicalized hand trajectory representation J, the train-time random rotation augmentation applies single random rotation matrix the whole canon- icalized hand trajectories. The same random rotation matrix, denoted as Rma. is added to the hand- object spatial representation as well. It is used transform the canonicalized object position, normal, and the hand-object offset vector:
Rend ((ox tx R Rmd ny R2 Rend, {he o+ R Rhe E J+)).
(25)
Similarly, the same random rotation matrix Rynd is used to transform the object velocity vector from vs in the temporal representation T to v
C.6 COMPLEXITY AND RUNNING TIME DISCUSSION
Denote the number of hand keypoints [J]. the number of generalized contact points [P], the complexity, the average inference time, and the number of forward diffusion steps for each denoising stage during inference summarized in Table 8.
34

---

Published as a conference paper at ICLR 2024
Table 8: Complexity and running time during the inference time.
MotionDiff
SpatialDiff
TemporalDiff
Average inference time (s)
0.52
16.61
7.04
Complexity
((J)
O(PII]T)
((P)
#Forword diffusion steps
400
200
100
35


Reviews:
[{"review_id": "xXb6dVWHrH", "reviewer": "Reviewer_C7Zj", "summary": "This paper proposes a HOI framework with a strong denoising capability to refine the hand trajectories in interaction sequences. The proposed GeneOH Diffusion method has two main key parts, one is a contact-centric HOI representation called GeneOH and the other is a domain-generalizable denoising scheme. Qualitative and Quantitative results show the methods outperform previous methods.", "questions": "- In Sec 3.2 Fitting for a hand mesh trajectory, the method denoises hand key points in the diffusion process and in the end optimizes to MANO parameters. How does the method ensure there is no penetration after going from keypoints to MANO mesh?\n- In Sec 4.1 Training dataset, why did the authors use different standard deviations to noise the MANO parameters for translation, rotation, and pose parameters Does the way to noise the training sets affect the learning?\n- In Sec 4.1 Evaluation dataset, what is the reason to use different noise on different test sets? Hope the authors give more explanation about it.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- The proposed method has strong spatial and temporal denoising capability and it generalizes well to novel HOI scenes. Other than existing methods, this paper explicitly parameterizes both the spatial and temporal relations between hands and objects.\n- The proposed progressive HOI denoising process is designed to use a separate stage to learn hand trajectory, and hand-object spatial and temporal relations collaboratively.\n- The proposed method outperforms the previous method TOCH on all 4 test sets by a large margin.", "weaknesses": "- In Fig 5 last column middle row, when the hand manipulating the scissors, the grasp does not make sense and the hand shown is penetrated into the scissors.\n- The method has the assumption that it requires accurate object pose trajectory to work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FuoGFrn5BP", "reviewer": "Reviewer_uUah", "summary": "- The authors introduce GeneOH Diffusion as a solution to address the challenge of denoising generalizable HOIs.\n- The authors develop GeneOH, a novel HOI representation that enhances generalization capabilities.\n- The authors propose a canonical denoising model for domain-generalizable denoising and a 3-stage progressive approach for HOI denoising.\n- This framework holds significant value for various downstream tasks. Experimental results on four test sets showcase the generalization ability of GeneOH Diffusion.", "questions": "- Some typos: Fig.3 caption: denoosing -> denoising\n- Why is it necessary to optimize $J^{stage2}$ using $\\mathcal{T}$ to obtain $J^{stage3}$ in the TemporalDiff stage instead of obtaining it through denoising? Have you conducted any ablation experiments to investigate this?\n- The authors' ablation experiments demonstrate that removing a certain stage of the diffusion model does not lead to a deterioration in the overall results in terms of IV or Penetration Depth. However, there is a significant degradation observed in HO Motion Consistency. The reviewer requests an explanation from the authors regarding this phenomenon.\n- Please consider citing: \"H2O: Two Hands Manipulating Objects for First Person Interaction Recognition\" in ICCV2021.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- On the technical side, the paper successfully employs diffusion models to achieve domain-generalizable denoising of hand-object interaction.\n- The progressive execution of denoising through multiple stages, as mentioned in the article, has merit and aligns with the representation capabilities of GeneOH.\n- The author offers a clear definition of a rational hand-object interaction trajectory.\n- The paper additionally conducts a comprehensive comparison with various methods using multiple test datasets.", "weaknesses": "- The description part of GeneOH can also be further optimized. It is recommended to begin with a summary of the description, including the considered features and the composition of the items. Subsequently, the symbol description can be expanded upon. It is advisable to avoid delving into excessive detail at the initial stages.\n- The authors provide experimental evidence demonstrating the generalizability of GeneOH Diffusion to the denoising of hand-object interaction sequences captured by real sensors. However, the majority of the experiments focus on adding known distribution noise to clean ground truth (GT) data and subsequently denoising it. This approach falls short in adequately analyzing the distinction between noise in real hand-object interaction trajectories and artificial noise, as well as evaluating the model's performance in real-world \"in-the-wild\" scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FvC4yfhP4e", "reviewer": "Reviewer_xeyt", "summary": "This paper tackles a complex issue - how to remove errors in hand-object interactions to make interaction sequences appear more realistic. This problem involves challenges such as unnatural hand positions, incorrect hand-object relationships, and the need to work well in various interaction scenarios and noise patterns. To address these challenges, the author proposes the framework called GeneOH Diffusion, which includes two key components: GeneOH, a method for representing interactions, and a denoising scheme that works well across different scenarios. The experiments show that the method outperforms others in various tests, suggesting its potential for a wide range of applications.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The writing of this paper is quite good. \n- The motivation is clear and the HOI denoising framework is well-designed.\n- The experiments are sufficient, where the framework shows promise for novel HOI scenarios.", "weaknesses": "I don’t have a big concern with this article since I’m not directly an expert in this area.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "klKpqaYPlW", "reviewer": "Reviewer_A9wU", "summary": "This paper tackles the problem of denoising HOI in images. The proposed method, GeneOH Diffusion, uses a novel approach that includes a contact-centric HOI representation named GeneOH and a new domain-generalizable denoising scheme. The contact-centric representation GeneOH parameterizes the HOI process, facilitating enhanced generalization across various HOI scenarios. The new denoising scheme consists of a canonical denoising model trained to project noisy data samples from a whitened noise space to a clean data manifold and a denoising via diffusion. The experiments on four benchmarks show the superior effectiveness of the proposed method.", "questions": "See the weaknesses part.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* GeneOH that encodes hand trajectories, hand-object spatial relations, and hand-object temporal relations to faithfully capture the interaction process.\n* Extensive experiments on four benchmarks with significant domain variations show the effectiveness of the method.\n* Codes are provided in the supplementary materials which is good for reproduction.", "weaknesses": "* The framework is complicated with three denoising stages. Complexity and running time discussion is missing in this paper.\n\n* Comparison and discussion with more recent works are missing, e.g. reference [1] is also a diffusion based HOI method.\n\n* Whitened noise space is constructed by diffusing the training data towards a random Gaussian noise. However, this technique is a common in diffusion model based applications.\n\n* No user studies or applications presented to demonstrate the practical utility of the denoised results. Quantitative metrics alone do not prove the outputs are naturally refined.\n\n\n[1] Diffusion-Guided Reconstruction of Everyday Hand-Object Interaction Clips", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)