DICE: End-to-end Deformation Capture of Hand-Face Interactions from a Single Image
Authors: Qingxuan Wu
Zhiyang Dou
Sirui Xu
Soshi Shimada
Chen Wang
Zhengming Yu
Yuan Liu
Cheng Lin
Zeyu Cao
Taku Komura
Vladislav Golyanik
Christian Theobalt
Wenping Wang
Lingjie Liu
Venue: ICLR 2025 Poster (2025)
Keywords: interaction
deformation
end-to-end
mesh recovery
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Reconstructing 3D hand-face interactions with deformations from a single image is a challenging yet crucial task with broad applications in AR, VR, and gaming. The challenges stem from self-occlusions during single-view hand-face interactions, diverse spatial relationships between hands and face, complex deformations, and the ambiguity of the single-view setting. The previous state-of-the-art, Decaf, employs a global fitting optimization guided by contact and deformation estimation networks trained on studio-collected data with 3D annotations. However, Decaf suffers from a time-consuming optimization process and limited generalization capability due to its reliance on 3D annotations of hand-face interaction data. To address these issues, we present DICE, the first end-to-end method for Deformation-aware hand-face Interaction reCovEry from a single image. DICE estimates the poses of hands and faces, contacts, and deformations simultaneously using a Transformer-based architecture. It features disentangling the regression of local deformation fields and global mesh vertex locations into two network branches, enhancing deformation and contact estimation for precise and robust hand-face mesh recovery. To improve generalizability, we propose a weakly-supervised training approach that augments the training set using in-the-wild images without 3D ground-truth annotations, employing the depths of 2D keypoints estimated by off-the-shelf models and adversarial priors of poses for supervision. Our experiments demonstrate that DICE achieves state-of-the-art performance on a standard benchmark and in-the- wild data in terms of accuracy and physical plausibility. Additionally, our method operates at an interactive rate (20 fps) on an Nvidia 4090 GPU, whereas Decaf requires more than 15 seconds for a single image. The code will be available at: https://github.com/Qingxuan-Wu/DICE.

Published as a conference paper at ICLR 2025
DICE: END-TO-END DEFORMATION CAPTURE OF HAND-FACE INTERACTIONS FROM A SINGLE IMAGE
Qingxuan Wu', Zhiyang Dou Sirui Xu², Soshi Shimada', Chen Wang's Zhengming Yu Yuan Liu?, Cheng Lin", Zeyu Cao , Taku Komura Vladislav Golyanik" Christian Theobalt', Wenping Wang", Lingjie Liu Li=
Universsiy of Pennsylvania, The University of Hong Kong
Unniversity of Illinois Urbana-Champaign, "Max Planck Institute for Informatics University of Cambridge, Texas A&M University, TTannGGP
ABSTRACT
Reconstructing 3D hand-face interactions with deformations from single image is a challenging yet crucial task with broad applications in AR, VR, and gaming. The challenges stem from self-occlusions during single-view hand-face interactions, diverse spatial relationships between hands and face, complex deformations, and the ambiguity of the single-view setting. The previous state-of-the-an Decaf, employs a global fitting optimization guided by contact and deformation estimation networks trained on studio-collected data with 3D annotations. However, Decaf suffers from a time-com consuming optimization process and limited generalization capability due to reliance on 3D annotations hand-face interaction data. To address these issues, we present DICE, the first end-to-end method for Deformation-awal hand-face Interaction reCovEry from a single image. DICE estimates the poses of hands and faces, contacts, and deformations simultaneously using Transformer-be -assed architecture. It features disentangling the regression of local deformation fields and global mesh vertex locations into two network branches, enhancing deformation and contact estimation for precise and robust hand-face face mesh recovery. To improve generalizability, we propose weakly-supervised training approach that augments the training set using in-the-wild images withoul 3D ground-truth annotations, employing the depths of 2D keypoints estimated by off- fff-the-shelf models and adversarial priors of poses for supervision. Our experiments demonstrate that DICE achieves caae-of-thee-r performance on standard benchmark and in-the- wild data in terms of accuracy and physical plausibility. Additionally, our method operates at an interactive rate (20 fps) on an Nvidia 4090 GPU, whereas Decaf requires more than 15 seconds for single image. The code will be available at: htttps://githiii com Qingxuunnnnnnnnnnnn Wu/DICE
INTRODUCTION
Hand-face interaction is common behavior observed up 800 times per day across all ages and genders (Spille et al.) 2021). Therefore, faithfully recovering hand-face interactions with plausible deformations is important task given its wide applications in AR/VR (Pumarola et al. 2018
Hu et al., 2017 Wei et al. 2019), character animation (Qin et al.] 2023 Zhao et al.] 2024). and
human behavior analysis (Liu al.) 2022 Guo et al., 2023]] Mueller al. 2019). Given the speed requirement of downstream applications like AR/VR, fast and accurate 3D reconstruction of hand-face interactions is highly desirable. However, several challenges make monocular hand-face deformation and interaction recovery particularly challenging: 1) selff-occlssoos involved in hand-face interaction, 2) the diversity of hand and face poses, contacts, and deformations, and 3) ambiguity in the single- view setting. Most existing methods (Rempe et al. 2020 Muller et al., (02T) only reconstruct hand (Romero et al. 2022) and face (Li et 2017) meshes, unified as a whole-body model (Loper et al. [023]|Pavlake et al. 2019), without capturing contacts and deformations. A seminal advance, Decaf (Shimada et al. 2023),, recovers hand-face interactions while accounting for both deformations and contacts. However, Decaf requires time-consuming optimization, which takes more than 15 seconds per image, rendering it unsuitable for interactive applications. Moreover, Decaf's 's iterative fitting process depends heavily on accurate initial estimates of hand and face keypoints, as well as contact
"Corresponding Authors

---

Published as a conference paper at ICLR 2025
Input Image
Front View
Side View
Input Image
Front View
Side View
T
2
(b)
(c)
Figure 1: Our method is the first end-to-end approach that captures hand-face interaction and deformation from a monocular image. Results are from (a) Decaf's 's validation dataset, (h) in-the-wild images, and (c) VR use cases.
points on their surfaces, which could fail when significant occlusion is present in the image (See Fig. 8). Additionally, Decaf cannot scale their training to fruitful hand-face interaction data in the wild, as they require 3D ground-truth annotations, such as contact labels and deformations that are not available from the in-the-wild data.
To tackle the issues above, we present DICE, the first end-to-end approach for Deformation aware hand-face Interaction reCovEry from monocular image. Our approach features three key designs: 1) Our Transformer-be based model leverages the attention mechanism capture the relationships between the hand and face. 2) Motivated by the global nature of pose and shape, well as the local nature of the deformation field and contact probabilities and face-we propose disentangling the regression of global geometry and local interaction into two network branches. We evaluate this approach to enhance the estimation of deformations and contacts while ensuring accurate and robust recovery of hand and face meshes. 3) Instead of directly regressing the hand and face parameters, we learn an intermediate paraamraecttic mesh representation. This representation is used regress the pose and shape parameters of the hand and face using neural network. Compared to directly regressing pose and shape parameters, which learns abstract parameters in highly non-linear space, predicting vertex positions in Euclidean space and then applying inverse-kinematics improves the reconstruction accuracy (Li et al.) 2021 2023c b). Combining all these contributions, our model achieves higher reconstruction accuracy than all previous regression- (Feng et al., 2021a Li Li al. 2017 Lin et al. (222a) and optimization- based (Shimada et al. 2023; Lugaresi et al. 2019 Li et al, 2017) methods. Additionally, by utilizing the neural eiverse-kinemattee network, our approach benefits from an animatable parametric representation of the hand and face, which can be readily utilized in downstream applications.
invariance to global transformations of the hand
peverse-kinematn
Despite containing rich 3D annotations, the existing benchmark dataset (Shimada et al. 22023) col- lected in a studio is still limited in the diversity of hand motions, facial expressions, and appearances. Training only such dataset limits the model's ability generalize to in-the-wild scenarios. To achieve robust and generalizable hand-face interaction and deformation recovery, we introduce weak-supervvsion training pipeline that utilizes in-the-wild images without the reliance on 3D annotations. To achieve this, our key insight is to leverage additional prior knowledge, such as depth supervision alongside 2D keypoint supervision, compensating for the absence of ground truth contact and deformation annotations. We leverage the robust depth prior provided by diffusion based monocular depth estimation model (Ke et al. 2024). which provides essential geometric information for accurate mesh recovery and captures spatial relationships critical for contact state and deformation estimation. As the task becomes highly ill ill-posed for in-the-wild images, we further employ pose priors of the hand and face by introducing hand and face parameter discriminators that learn rich hand and face motion priors from additional datasets on hand or face separately (Pan et al., 2002al Zim mermann al. 2019). By incorporating small set of real-world images alongside the Decaf dataset and leveraging our weak-supervisiion pipeline, we markedly enhance the accuracy and generalization capacity of our model.
As a result, our method achieves superior performance terms of accuracy, physical plausibility, inference speed, and generalizability. It surpasses all previous methods in accuracy both standard
2

---

Published as a conference paper at ICLR 2025
benchmarks and challenging wn-the-wild images. Fig. visualizes some results of our method. We conduct extensive experiments to validate our method. In summary, our contribution is three-fold:
We propose DICE, the first end-to-end learning- -ased approach that accurately recovers hand-face interactions and deformations from a single image.
We propose a novel veak--uupervied training scheme with depth supervision on keypoints to augment the Decaf data distribution with diverse real-world data distribution, significantly improving the generalization ability.
DICE achieves superior reconstruction quality compared to baseline methods while running at an interactive rate (20fps).
2 RELATED WORK
Extensive efforts have been made to recover meshes from monocular images, including human
bodies (Bogo et al., 2016); Moon & Lee| 2020; Li et al., 2021; Cai et al. 2024] Contributors Xic
ct al. 2022, Wang & Danilldds] 2023) Wang et al. 2023b
Lin et al. (222
Kanazawa et al. 2018)
Cai et al. [2022] Zhang et al.] [022bb Feng et al., 2023
Li et al. [202cc] Wang et al. 2023a Dou
et al. |023b; Cho et al. |022 Huang et al. 2022bb
Lin et al.]
202 hands (Rong et al. 2221
Moon et al. 2020 2024 Moon
2023 Oh et al. 2023 Park et al.
2022: Yang et al. 2021 2022b; Li
et al.
2023d Yu et al. 2023), and faces (Feng et al. [222bb]
2018
Wood et al. 2022
Daneček et al.
2022 Zielonka et al. 2022 Chai et al. [023] Zhang et al.
2023c
Otto et al.
2023
He et al. 2023
Chatziagapi & Samaras 2023] Kumar et al. 2023]] Li et al.
2023a).
This also includes recovering the
surrounding environments (Clever et al. [022]] Huang et al.
2022a
Hassan et al.
2019 202T; Zhang
et al. 2020b Li et al. 2022b] Zhang et al. 2021c; Shimada et al.
2022
Luo et [022| Weng &
Yeung (2021) and interacting objects YYng et al.
2022a
Zhang
al.]
2020a| Pham et al. [2017]
Tsoli & Argyros 2018: Hampali et al.] 2020 Tekin et al.
2019;
Zhang et al. 2020a| Grady et al.
2021:
Pokhariya et al. [2023] Hasson et al. 2019 Ye et al.
2022 Chen et al. 2023) 2021 Liu et al.
2021
Corona et al., (2020) while reconstructing the mesh.
The acquired versatile behaviors play
crucial role in various applications, including motion generation (Tevet et al. [2022] Peng ct al. 2022
Pan et al.] 2023b Guo et al. 2022 Wang et al., 2022a| Xu et al., [0233
Lin et al. 2024 Zhou
et al. a [0233] Wan et al. 2023a] Peng et al.] 2022 | Dou et af.) 2023a] Wan et al. 2023b)
augmented
reality (AR), virtual reality (VR), and human behavior analysis (Zhang et al. 2023a Yang et al.)
2024, Zhang et al. 2024| 2023b Guo et al. 2023 Liu et al. (2022). In the following, we mainly review the related works on hand, face and full-body mesh recovery.
3D Interacting Hands Recovery. Recent advancements have markedly enhanced the capture and recovery of 3D hand interactions. Early studies have achieved reconstruction of 3D hand-hand interac- tions utilizing a fitting framework, employing resources such as RGBD sequences (Oikonomidis et al.) 2012). hand segmentation maps (Mueller et al. 2019), and dense matching maps (Wang et al. 2020). The introduction of large-scale datasets for interacting hands (Moon et al.) 2020 2024) has motivated the development of regression- based approaches. Notably, these include regressing 3D interacting hand directly from monocular RGB images (Rong et al. 2021 Moon| 2023 Zhang et al. 222
Li et al. 2022a; Zuo et al., 2023). Additionally, research has extended to recovering interactions
between hands and various objects in the environment, including rigid (Cao al. |02 Grady
cct al. 2021 Liu et al. 2021 Tekin et al. [011 Fan et al. 2024 Ye et al., 2023b a). articulated (an
al. 2023). ) and deformable (Trettchk al. 2023) objects. Following Shhiada et al. (2023)
work distinguishes itself by introducing hand interactions with a deformable face, characterized
by its non-uniform stiffness-2 significant difference from conventional deformable models. This innovation presents unique challenges in accurately modeling interactions.
3D Human Face Recovery. Research in human face recovery encompasses both optimization- based Aldrian & Smith Thies Thies et al.) 2016) and regression-based (Feng et al., 2018 Sanyal al. 2019) methodologies. Beyond mere geometry reconstruction, recent approaches have evolved
to incorporate training networks with the integration of differentiable renderers (Feng et al., 2021b
Zielonka et al. 2022. Zheng et al., 2022 Wang et al.) 20226) Cho et al.] (002)). These methods estimate variables such as lighting. albedo, and normals to generate facial images and compare them with the monocular input. However, a significant limitation in much of the existing literature
is the neglect of the face's deformable nature and hand-face interactions. Decaf (Shimada et al. 2023) represents a pivotal development in this area, attempting to model the complex mimicry of musculature and the underlying skull anatomy through optimization techniques. In contrast, our work introduces a regression-basee end-to-end method for efficient mroblem-solving setting a new benchmark in the field.
3

---

Published as a conference paper at ICLR 2025
Depth Supervision on Keypoints
Interatiionett
Pseudo-GT Depth Generation
Feature Sharing
Dfftrrntiable Renderer
Apply Deform
Aggregate
Head ICNet
Mildel
CNN
MeshNet
ItNee
Real/Fake?
Head D
Hand
Real/Fake?
M Hand Distrbbtiio
Head Distribution
Predicted
Predicted
Head Mesh Features
FLAME Parameters (8) (8/-pase -exp))
Head Mush
Contact Labell Cy,
Image Features
Hand Mesh Features Vi
MANO Parameters (9s, Pa)
Hand Mesc
Predicted
Deform Field DD
Predicted
Figure 2: Overview of the proposed DICE framework. The input image is first fed to CNN to extract feature map, which is then passed to the Transformer-based encoders for mesh and interaction, ie.. MeshNet and InteractionNet. MeshNet extracts hand and face mesh features, which then used the Inverse Kinematics models (IKNets) predict pose and shape parameters
drive FLAME (Li al. 2017) and MANO (Romero ct al. (2022) models. InteractionNet predicts per per-vertex hand-face contact probabilitic and face deformation fields from the feature map, where the latter is applied to the face mesh output by the FLAME model. To improve the generalization capability, introduce weeakly-supervice detection models (Lugaresi al. 2019 Bulat Tzimiropoulos) 2017) and depth estimation models (Ke al., 2024) to provide depth supervision on keypoints. In addition, we use face and hand discriminators to constrain the distribution of parameters regressed by IKNets.
training scheme using off-the-sheff 2D keypoint
3D Full-Body Recovery. The task of monocular human pose and shape estimation involves recon-
structing 3D human body from single image. Optimization-basel approaches (Bogo et al. 2016
Pavlakos et al.
2019 Shi et al. 2023 Rempe al. (02T) employ the SMPI. model (Loper et al.
2023). fitting
2D keypoints detected within the image. Conversely, regression basodddddaaaedoooooo methods LLi
ct al. 2021 Lassner et al. 2017 Kocabas et al.) 2021 Kanazawa et al. 2018 Feng et al. 202 la
Fang et al.
2021 Lin et al. 2023 Cai et al. 2024 Feng al. 2023) leverage deep neural networks
to directly infer the pose and shape parameters of the SMPL model. Hybrid methods (Kolotouros)
al.
2019a) integrate both optimization and regression techniques, enhancing 3D model supervision.
Distinct from these approaches, we follow parametric methods (Li et al. 2021 Cai et al.) 2024] Kanazawa et al. 2018: Bogo al. (0166 due its flexibility for animation purposes. Unlike most research in this domain, which primarily concentrates on the main body with only rough estima- tions of hands and face, our methodology uniquely accounts for detailed interactions between these components.
METHOD
Problem Formulation. Following Decaf (Shimada al. 2023), we adopt the FLAME (Li et al. 2017) and MANO (Romero et al. 2022) parametric models for hand and face. Given single RGB image I IR 22222XX the objective of this task is to reconstruct the vertices of a hand mesh VH I. 778xx and a face mesh VF و e IR 5023x3 along with capturing the face deformation vectors D 5023x3 resulting from hand-face interaction and its non-rigid nature. Additionally, estimate
per-vertex contact probabilities of hand Cи E 777 and face Cr E 55023
3.1 TRANSFORMER BASED HAND-FACE INTERACTION RECOVERY
Our model incorporates two-branch Transformer architecture and integrates inverse-kinemant models, specifically, MeshNet, InteractionNet. and IKNets. A differentiable renderer (Ravi et al. 2020) is used compute depth maps from the predicted mesh for depth supervision, while the hand and face discriminators are used as priors for constraining the hand and face poses; See Fig. 2 for an overview.
4

---

Published as a conference paper at ICLR 2025
Given monocular RGB image I. extract a feature map X1 E R XxC, x Following Lin et al. 2202aa b). we flatten the image feature maps and upsample the H W feature maps to N feature maps, corresponding each keypoint and downsampled vertex of both hand and face. The feature maps F' E N are then concatenated with the downsampled hand and face vertex and keypoint coordinates of dimension N x 3, with the pose set to the mean pose, serving as positional encodings. This results in the final feature map E R (RNx(C+3)) To model the vertex-vertex interactions, we mask the feature maps F for a randomly selected subset of vertices.
a pretrained HRNet- W64 (Sun al. 2019) backbone to
Once the feature map F is obtained, it is fed into MeshNet and InteractionNet, which handle the regression of mesh vertices and the deformation field separately. This decomposition is motivated by their semantic differences: mesh contains global features, whereas deformation vectors and contact states are localized features, i.e., invariant the global transformations of the hand and face. Thus, MeshNet takes the feature map F as input and regresses the unrefined vertex positions of hand Vn and face V InteractionNet, on the other hand, predicts the 3D deformation field for each face vertex, along with the contact labels for each hand and face vertex, Cи and Cy, respectively. Note the contacts and deformations are regressed within the same encoder to model their causal relationship: the contacts cause the deformations. We validate our design in Sec. [.4
Next, instead of directly using the unrefined hand and face vertices VHH and V our method takes these vertices as input to regress the pose and shape of their respective parametric models (Li et al. 2017 Romero et al. (2022). This is achieved by a neural inverse kinematics model, named IKNet, following Kolotouros al. (2019b). The IKNet takes the unrefined hand and face vertices V'H and F as inputs and predicts their pose, shape, and expression parameters (On, Bb) for hand, porep Br. exe)) for face, along with the root position and orientation for hand (t,, rv)) and face respectively. Afterward, we use the predicted parameters to first obtain the hand vertices V "
(ty, r (tg.ry). and undeformed face vertices Vi. Then, we apply the deformation D predicted by the InteractionNet on Vi to get the final deformed face Vp. Utilizing parametric dawward-kinemate inverse-kinematic models offer several advantages: first, it enables readily animatable meshes for downstream applications; second, compared to non-parametric regression methods, where meshes typically contain artifacts such as spikes (Lin et al. (221aa Cho et al.] 2022, Lin et al., 2021b), this approach significantly improves mesh quality; third, the compact parameter space allows for more effective discriminator, which will be discussed in the following section.
and neural
3.2 SUPERVSS TRAINING SCHEME
Although the aforementioned benchmark, Decaf (Shimada et al., 2023), accurately captures hand, face, self-contact, and deformations, it consists of only eight subjects and recorded in green-screen studio. Thus, training a model only with the Decaf dataset limits its generalization capability to in-the-wild images that exhibit far more complex and diverse human identities, hand poses, and face poses.
To further enhance the generalization capability, we train our model with 500 diverse in in-the-wy tild images of hand-face interaction collected from the internet without the reliance on the 3D ground truth annotations. First, we use 2D hand and face keypoints detected by Lugaresi et al.) (2019) and
Bulat & Tzimiropoulos (017)) as pseudo-ground-tng a diffusion-based monocular depth estimator pre-trained on large amount of images to generate 2D affine-invarian depth maps for depth supervision (see Eq. The depth supervision provides a strong depth prior, which guides the spatial relationship between hand and face meshes, promoting accurate modeling of hand-face interaction. We first use a differentiable rasterizer (Ravi et al.) 2020) to compute a depth map from the predicted hand and face meshes. We use a depth loss to measure the difference between the depths of the hand and face keypoints and their corresponding points on the predicted depth map, providing supervision. This porpointtookeyporri correspondence enables accurate depth supervision even when the rendered hand/face mesh and the ground-trute meshes are misaligned. Moreover, we train adversarial priors on the hand and face parameter space on multiple hand and face motion datasets: the face-only RenderMe-360 (Pan al., 2023a), the hand-only FreiHand (Zimmermann et al. (011)), and Decaf (Shimada et al., 2023). This ensures the plausibility of generated face and hand poses and shapes while allowing for flexible poses and shapes beyond the Decaf data distribution to handle in-the-wild cases. The overall weak weak-supervision pipeline significantly enhances our model's 's generalization capability and robustness, which we investigate in Sec. 4.4
Then, we use Marigold (Ke et al. 2024).
5

---

Published as a conference paper at ICLR 2025
3.3 LOss FUNCTIONS
Mesh losses G For richly annotated Decaf dataset (Shimada et al. 2023 we employ an L1 loss for 3D keypoints, 3D vertices, and 2D reprojected keypoints, comparing them against their respective ground-truths, following common practice human and hand mesh recovery (Lin et al.) 2021 Cho 2022 Dou et al. 2023b). We further apply an Ln loss f on the estimated hand and face pose, shape, and facial expression against the ground-truth parameters. For in-the-wild data, only the 2D reprojected keypoints are supervised, as they are the only type with corresponding ground truth.
et al.
Interaction losses Linteraction: Similar to Shimada et al. (023), we impose Chamfer Distance losses to promote touch for predicted contact vertices and discourage collision. We also introduce a binary cross-entropy loss to supervise contact labels and a deform loss with adaptive weighting mechanism to supervise deform vectors. For mn-the-widd data, we also impose touch and collision losses since they do not require annotations.
Adversarial loss Luiv: The adversarial losses applied to the predicted hand and face parameters for in-the-will wwild data to constrain their parameter space, and for Decaf data to facilitate the training of the discriminators. The adversarial loss is given by:
Ladv (E) Egyope [log (1- DF(E())) + Ee [log (1 DH((())))
(1)
The losses for the hand and face discriminators are given by:
Lade (Dp) (Bor
~pe [log (1 (=(((((()) BB, ~um [log (Dp(O,))))
and
Ludv (DH) -(Eon~p Dog (1 DH (E(())) + - Dog (D ( (0x))))
where E jointly denotes the image backbone, the mesh encoder and the parameter regressor, PE denotes the output distribution of E., Pdata denotes the data distribution of the motion datasets, 0, (Or-pose, Gr, 0t-xxp) OH (On: Sb).
Depth loss Laeph: To provide pseudo-3D hand and face keypoints supervision for in-the-wild data, we use a modified SILog Loss (Eigen et al. 20014), an affine-invariant depth loss as our depth supervision Laspth Formally, let KD denote the pseudo-ground ground-truth affine- invarant depth of the face and hand keypoints, and KD denote the rendered depth for the keypoints,
Var (log( (KD + + log(A + =))] 1/2
Laepth
where Var the standard variance operator and & 10-7
Overall, our loss for the mesh and interaction networks is formulated by
Amesh mmssh + Xinteraction Lintemation + Andv Lady + >depth
(5)
where Amesh 12.5, Aintoraction 5, Adepth 2.5, Andv 1 for all the experiments in the paper; See more details in Appendix C
4 EXPERIMENTAL RESULTS
4.1 DATASETS AND METRICS
Datasets. We employ Decaf (Shimada et al. 2023) for reconstructing 3D face and hand interactions with deformations, along with the in-the-wild dataset we collected containing 500 images. We use the shape, pose, and expression data of hands and faces from Decaf (Shimada et al. 2023), RenderMe-360 (Pan et al.) 2023a). and FreiHand (Zimmermann al. 2019) for training the adversarial priors. We use the training set of the aforementioned datasets for network training. We use the official split from Decaf to separate the training and testing sets, and select a few in-the-wild images for the test set to perform qualitative visualizations.
Metrics. We adopt commonly-used metrics for mesh recovery accuracy following Kanazawa et al. (2018); Lin et al. (202 Dou et al. (2023b); Cho et al. (0022): 1) Mean Per-Joint Position Error (MPJPE): the average Euclidean distance between predicted keypoints and ground-truth keypoints. 2) PAMPJPE: MPJPE after Procrustes Analysis (PA) alignment. 3) Per Vertex Error: per vertex
error (PVE) with translation. Following Decaf (Shimada et al. 2023), we use the following metrics measure the plausibility: 4) Collision Distance (Col. Dist.): the average collision distances

---

Published as a conference paper at ICLR 2025
Reconstruction (Face Hand)
Roconstruction (Face only)
Contact (Face)
Contact (Hand)
Input Imuge
Decaf
In In-the-wild wild
Figure 3: Qualitative results of hand-face interaction, deformation, and contact recovery by DICE on Decaf and in-the-wild images. In contact visualizations. deeper color indicates higher contact probability.
vertices and frames; 5) Non-Collision Ratio (Non. Col.): the proportion of frames without hand-face collisions, 6) Touchness Ratio (Touchness): the ratio of hand-face contacts among ground truth contacting frames; 7) F-Score: the harmonic mean of Non-Co Collision Ratio and Touchness Ratio. Note that F-Score measures Touchness and Non-Collision Ratio whole, which is metric of overall physical plausibility, whereas Non-Collision Ratio or Touchness are meaningless when considered individually.
4.2 IMPLEMENTATION DETAILS
We train MeshNet, InteractionNet, and IKNet, along with the face and hand discriminators using AdamW (Loshchilov 2017) optimizers, each with a learning rate of 6 x 10-4, and learning rate optimized in alternating manner.
decay of 1 10 The generator and discriminator networks Our batch size is set to 16 during the training stage. The training takes 40 epochs, totalling 48 hours. The model is trained and evaluated on 8 Nvidia A6000 GPUs with AMD 128-core CPU. Inference times are calculated on single Nvidia A6000 GPU.
4.3 PERFORMANCE ON HAND-FACE INTERACTION AND DEFORMATION RECOVERY
We compare our method with the following: 1) Benchmark: the baseline Lugaresi et al. 2019 Li al. 2017) introduced in Decaf (Shimada et al., 2023); 2) Decaf (Shimada et al. (023): an optimization-bant
method for hand-face interaction and deformation recovery. 3) PIXIE (whole- body) (Feng et al.) 2021a): representative model for full-body recovery, ineluding the hand and face, introduced in Decaf. 4) PIXIE (hand+face) (Feng et al. 2021a): optimization- based variant of PIXIE, introduced in Decaf. For regression-baseer based methods,
we are dealing with relatively new task, there are few readily available baselines. To facilitate comparison, we adapt the following regression-basee models from related tasks: 5) METRO (Lin et al. 2021a): ; representative work in human body/hand mesh recovery. We adapt METRO to predict both hand and face meshes, adding extra output heads to predict contact and deformation. 6) PIXIE-R (Feng et al.) 2021a): Adapted PIXIE, using the same backbone and hand and face branches but trained with losses from DICE. 7) FastMETRC (single-target) (Cho et al., 2022): Another representative work human and hand mesh recovery. We adapt two independent FastMETROs, one for estimating hand mesh vertices and contact, and the other for estimating face mesh deformation, and contact. Here, the word single-targe means each FastMETRO considers hand and face individually. with no information exchange. This model is trained using the same hyperparameter, loss, and optimizer as DICE, on the Decaf (Shimadal al. 2023) dataset.
7

---

Published as a conference paper at ICLR 2025
Input Image
Ours
Decaf
PIXIE
METRO
Decaf
25
2
u
In-the-wild
Figure 4: Qualitative comparison of DICE, Decaf (Shimada et al. 2023 PIXIE (Feng al. 2021a) (whole-
body version), METRO* (Lin al. 2021b) on Decaf validation set and in-the-wild images. Our method achieves superior reconstruction accuracy and plausibility in the Decaf (Shimada al. (2023) dataset, especially generalizing well to difficult in-the-wild actions unseen in Decaf compared to all baselines.
4.3. QUANTITATIVE EVALUATIONS
Reconstruction Accuracy. In Tab. our method surpasses all baseline methods in terms of reconstruction accuracy, achieving 7.5% reduction per-vertex error compared the current state- of-the-art, Decaf. Note that our method is regression babessd and allows inference at an interactive rate, while Decaf (Shimada et al. 2023) uses cumbersome test-time optimization process, taking more than 200x more time per image. Decaf also requires using temporal information in successive frames, while our method only uses a single frame. Our method shows a 30% reduction reconstruction error compared to the modified METRO baseline, and up to 79% reduction compared to other end-to-end baselines. Notably, our method achieves 27% MPVE reduction compared to the PIXIE-R baseline which uses the same mesh and interaction losses method, demonstrating the superiority of our network design and weak-superviin training scheme. Our method is also more accurate than another end-to-end baseline, FTStMETRO.
8

---

Published as a conference paper at ICLR 2025
Table 1: Comparison of hand-face interaction and deformation recovery Decaf.
Recomsmaction Error
Physies Plausibility Metrics
Running Time
Methods
Type
PVE3 MPIPEJ PAMPIPE, Col. Dist. J. Non. Col. I Touchhesss | F-Score
(per image; SX
Compartson between DICE and optimization-
methods
Docaf (Shimada et
2023
9.65
1.03
83.6
96.6
89.6
19.59
Benchmark Luggarest
2019
Ti ct al.
2017
17.7
19.3
64.2
73.2
68.4
16.40
PIXIE (hand-tace) Fong d
26.3
7:04
75.9
75.1
75.5
DICE (Ours)
R
8.32
9.95
7.27
0.16
06.6
79.9
72.7
0,088
Comparison between DICE regression baved methods
PIXIE (whele bodv) Feng et al. 2021a]
R
39.7
0.11
97.1
51.8
67.6
0,070
PIXIE-R (Feng et 2023
22.0
21.2
62.6
83.0
72.0
0.070
METRO (nind-lace) (Lan et al. 20
R
11.8
15.4
11.9
0.08
80.7
54.8
65.2
0.103
FastMETRO* (single-tirge))
et
R
11.8
9.41
82.2
55.5
68.2
0.130
DICE (Ours)
R 8.32
9.95
7.27
0.16
06.6
79.9
72.7
0.088
parametric version. the bead to the origin. bold denotes the best result in comparison group. Note our method operates at interactive rate (20 0.049s per image) on an Nvidia 4090 GPU, Here we report the ruatime performance on an A6000 GPU fair comparison.
denose optinization- and regression-basse based mothods, respectively, calculated after translating the couter of
Table 2: Comparison of hand-face contact estimation on Decaf.
Method
F-score
Precision
Recall
Accuracyt
Decaf (face) Shimads al 2023
0.57
0.69
0.49
0.99
Decaf (han))
Stamada 4 al. 2023
0:47
0.62
0.39
0.98
DICE (face)
0.61
0.84
0.57
1.00
DICE (hand)
0,50
0.55
0.45
0.98
Plausibility. In terms of overall physical plausibility (F-Score), our method is the best among all regression-bad methods: PIXIE (whole-body). PIXIE-R, METRO. and FastMETRO. On the other hand, while some optimization based methods (Decaf and PIXIE (hand+face) have higher overall plausibility (F-Score) compared to DICE, this due to their test-time optimization, which iteratively adjusts the relative positioning of hand and face. Thus, they are much more computationally intensive than our regression- based method. With a highly efficient end-to-end inference scheme, DICE still outperformed an optimization-base method (Benchmark) on F-Score.
Contact Estimation. The contact estimation metrics (accuracy, precision, recall) are calculated by the predicted per-vertex contact probabilities against the respective 0-1 contact ground truths. In Tab. 2. DICE achieves superior contact estimation performance on the Decaf dataset, surpassing previous work (Shimada et al. 2023) in F-Score for both face and hand contacts. Here F-score provides comprehensive measure of both precision and recall ratio combined. These two metrics involve a trade-off: focusing solely on precision may lead to decrease in recall, and vice versa. Balancing this trade-off, the F-score offers a more meaningfur evaluation of contact estimation.
4.3.2 QUALITATIVE EVALUATIONS
As discussed Sec. 3.2 the Decaf (Shimada al. 2023) dataset is collected in indoor environment with a green screen, which doesn't it reflect the complex environment where real-world hand-fac interactions occur. Therefore, a model only trained with the Decaf dataset might have generalization issues when tested on in-the-wild data. Fig. 4 supports this claim by demonstrating our model's s superior generalization performance on in-the-wild data with unseen identity and pose. On the other hand, Decaf's ' reconstruction suffers from self-collision and incorrect hand-face relationship. PIXIE and METRO reconstruct inaccurate hand poses and often demonstrates implausible non- touching artefacts. As shown in Fig. 3 our method faithfully reconstructs hand-face interaction and deformation and accurately labels the contact areas.
4.4 ABLATION STUDY
Network Design. In Tab. adopting the two-branch architecture, which separates deformation and interaction estimation from mesh vertices regression, improves both accuracy and plausibility.
In-the-wild data. As shown in Tab. 3 adding weak-supervision training and in-the-wild data for DICE training improves all reconstruction error metrics (PVE*, MPJPE, PAMPJPE) while maintaining high plausibility (F-Score). We deem that the slight decrease in F-Score could mainly be attributed to the difference in distribution between the studio-collected Decaf (Shimada et al. 2023) and in-the-wild data. This is because the limited pose and identity distribution of the Decaf training dataset may cause the model to overfit, and the inclusion of in-the-wil images out of the Decaf data distribution effectively improves the generalization capability of DICE.

---

Published as a conference paper at ICLR 2025
Table Comparison of hand-face interaction and deformation recovery on Decaf. Bold denotes the best result.
Methods
PVE* MPJPE] PAMPJPE, F-Score f
DICE (single branch)
9.29
11.6
5.51
69.3
DICE (wo. in-the-wild data)
8.93
11.0
7.50
73.3
DICE (W.O. supervision V,,, Vir)
12.2
14.4
11.1
70.7
DICE (w.o. Lapth)
15.6
19.5
13.7
64.2
DICE (w.o. Lprom))
10.3
12.8
10.4
61.7
DICE (w.o, Levv)
11.1
14.2
10.4
69.8
DICE (Full)
8.32
9.95
7.27
72.7
Unrefined Features Supervision. Regressing the unrefined head and hand mesh features V'p, V and then perform inverse kinematics to regress the parametric mesh improves plausibility and aceuracy, compared to directly estimating the face and hand parameters.
Depth Supervision. Although depth supervision is only applied to on-the-wild data, as shown in Tab. 3] removing it also significantly degrades performance on the Decaf validation set. Without depth loss, wrong predictions in depth are not penalized for in-the-wild data, introducing noise in the training process, and resulting in erroneous predictions on the Decaf dataset. As shown in Appendix Fig. 7 the absence of depth supervision introduces ambiguity in the z-direction, resulting in artifacts such as self-collision.
Parameter Supervision. Supervising parameters directly, in addition to the indirect supervision of parameters by the mesh losses, improves both plausibility and accuracy. This is because direct parameter supervision eliminates ambiguity, preventing the network from converging to alternative parameter combinations that produce incorrect meshes that appear geometrically similar, i.e. with small vertex loss, to the target but incorrect in their underlying structure, such as pose shape.
The adversarial prior incorporates diverse but realistic pose and shape distribution beyond Decaf (Shimada et al.) 2023), ensuring the reality of regressed mesh while allowing for generalization. As shown in Tab. introducing adversarial supervision improves the accuracy and physical plausibility.
Adversarial Prior.
4.5
LIMITATIONS AND FUTURE WORKS
While our method achieves SotA accuracy on the Decaf (Shimada et al. 2023) dataset and generalizes well to unseen scenes and in-the-wild cases, it still encounters failure cases when the hand-pose interactions are extremely challenging and have severe occlusions (see Appendix D.2). Moreover, despite method effectively recovering hand and face meshes with visually plausible face deforma- tions, there remains room for improvement in deformation accuracy and physical plausibility. Hand deformations could also be considered in future work for more realistic reconstructions. In the future, physics-based simulation (Hu et al. 2018 Li et al. 2020: Hu et al., 2019; Han et al. 2019 Lin al. 2022 Huang et al.) (224) can be used as a stronger prior, producing more physically accurate estimations. In this paper, although we found using 500 in-the-wild images significantly improves the model's generalization ability, scaling up a larger amount of in-the-wild data, on the order of millions or billions, would further enhance performance, which we will study in future work.
5 CONCLUSION
In this work, we present DICE, the first end-to-end approach for reconstructing 3D hand and face interaction with deformation from monocular images. Our approach features a two-branch transformer structure, MeshNet, and InteractionNet, to model local deform field and global mesh geometry. An inverse-kinematic model, IKNet, is used to output the animatable parametric hand and face meshes. We also proposed novel weak-suuprvision training pipeline, using a small amount of in-the-wild images and supervising with depth prior and an adversarial loss to provide pose priors. Benefitting from our network design and training scheme, DICE demonstrates state-of-the accuracy and plausibility, compared with all previous methods. Mcanwhile, method achieves a fast inference speed (20 fps), allowing for more downstream interactive applications. In addition to strong performance on the standard benchmark, DICE also achieves superior generalization performance on in-the-wild data.
ACKNOWLEDGMENTS
This work is partly supported by the Research Grant Council of Hong Kong (Ref: 17210222). the Innovation and Technology Commission of Hong Kong under the ITSP-Platform grants (Ref: ITS/319/21P,, ITS/335/23FP) and the InnoHK initiative (TransGP project). The research work was in part conducted the JC STEM Lab of Robotics for Soft Materials funded by The Hong Kong Jockey Club Charities Trust.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Oswald Aldrian and William AP Smith. Inverse rendering of faces with a 3d morphable model. IEEE transactions on patiern analysis and machine intelligence, 35(5): 2012.
Federica Bogo, Angjoo Kanazawa, Christoph Lassner, Peter Gehler, Javier Romero, and Michael J Black. Keep smpl: Automatic estimation of 3d human pose and shape from a single image. In European conference computer vision, pp. 561-578. Springer, 2016.
Adrian Bulat and Georgios Tzimiropoulos. How far are we from solving the 2d & 3d face alignment problem? (and dataset of 230,000 3d facial landmarks). International Conference on Computer Vision, 2017.
Zhongang Cai, Daxuan Ren, Ailing Zeng, Zhengyu Lin, Tao Yu, Wenjia Wang, Xiangyu Fan, Yang Gao. Yifan Yu, Liang Pan, et al. Humman: Multi-modal 4d human dataset for versatile sensing and modeling. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part VII, pp. 557-577. Springer, 2022.
Zhongang Cai, Wanqi Yin, Ailing Zeng, Chen Wei, Qingping Sun, Wang Yanjun, Hui En Pang. Haiyi Mei, Mingyuan Zhang, Lei Zhang, et al. Smpler-x: Scaling up expressive human pose and shape estimation. Advances Neural Information Processing Systems, 36, 2024.
Zhe Cao, Ilija Radosavovic, Angjoo Kanazawa, and Jitendra Malik. Reconstructing hand-object interactions in the wild. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12417-12426. 2021.
Zenghao Chai, Tianke Zhang, Tianyu He, Xu Tan, Tadas Baltrusaitis, HsiangTao Wu, Runnan Li, Sheng Zhao, Chun Yuan, and Jiang Bian. Hiface: High-fidelity 3d face reconstruction by learning static and dynamic details. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 9087-9098 2023.
Aggelina Chatziagapi and Dimitris Samaras. Avface: Towards detailed audio-visual 4d face re- construction. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 16878-16889, 2023.
Jiayi Chen, Mi Yan, Jiazhao Zhang. Yinzhen Xu, Xiaolong Li, Yijia Weng, Li Yi, Shuran Song, and He Wang. Tracking and reconstructing hand object interactions from point cloud sequences in the wild. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 304-312, 2023.
Yujin Chen, Zhigang Tu, Di Kang, Ruizhi Chen, Linchao Bao, Zhengyou Zhang, and Junsong Yuan. Joint hand-object 3d reconstruction from single image with cross-branch feature fusion. IEEE Transactions on Image Processing, 30:4008 4021, 2021.
Junhyeong Cho, Kim Youwang, and Tae-Hyun Oh. Cross-attention of disentangled modalities for 3d human mesh recovery with transformers. In European Conference on Computer Vision, pp. 342-359. Springer, 2022.
Henry M Clever, Patrick L Grady, Greg Turk, and Charles Kemp. Bodypressure-inferiin body pose and contact pressure from a depth image. IEEE Transactions Pattern Analysis and Machine Intelligence, (5(1):137-155) 2022.
MMHuman3D Contributors. Openmmlab 3d human parametric model toolbox and benchmark.
Enric Corona, Albert Pumarola, Guillem Alenya, Francese Moreno-Noguer, and Grégory Rogez. Ganhand: Predicting human grasp affordances in multi-object scenes. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5031-5041. 2020.
Radek Daněček, Michael Black, and Timo Bolkart. Emoca: Emotion driven monocular face capture and animation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20311-20322, 2022.
Zhiyang Dou, Xuelin Chen, Qingnan Fan, Taku Komura, and Wenping Wang. C- ase: Learning conditional adversarial skill embeddings for physics- characters. In SIGGRAPH Asia 2023 Conference Papers, pp. 1-11, 2023a.
11

---

Published as a conference paper at ICLR 2025
Zhiyang Dou, Qingxuan Wu, Cheng Lin, Zeyu Cao, Qiangqiang Wu, Weilin Wan, Taku Komura, and Wenping Wang. Tore: Token reduction for efficient human mesh recovery with transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15143-15155, 2023b.
David Eigen, Christian Puhrsch, and Rob Fergus. Depth map prediction from single image using multi-scale deep network. Advances in neural information processing systems, 27, 2014.
Zicong Fan, Omid Taheri, Dimitrios Tzionas, Muhammed Kocabas, Manuel Kaufmann, Michael J. Black, and Otmar Hilliges. ARCTIC: dataset for dexterous bimanual hand-object manipulation. In Proceedings IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2023.
Zicong Fan, Maria Parelli, Maria Eleni Kadoglou, Muhammed Kocabas, Xu Chen, Michael J Black, and Ounar Hilliges. HOLD: category-agnoshe 3d reconstruction of interacting hands and objects from video. Proceedings IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Qi Fang, Qing Shuai, Junting Dong, Hujun Bao, and Xiaowei Zhou. Reconstructing 3d human pose by watching humans in the mirror. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12814-12823, 2021.
Yao Feng, Fan Wu, Xiaohu Shao, Yanfeng Wang, and Xi Zhou. Joint 3d face reconstruction and dense alignment with position map regression network. In Proceedings of the European conference on computer vision (ECCV), pp. 534-551, 2018.
Yao Feng, Vasileios Choutas, Timo Bolkart, Dimitrios Tzionas, and Michael J Black. Collaborative regression of expressive bodies using moderation. In 2021 International Conference 3D Vision (3DV), pp. 792-804, 2021a.
Yao Feng, Haiwen Feng, Michael J Black, and Timo Bolkart. Learning animatable detailed 3d face model from in-the-wild images. ACM Transactions on Graphics (ToG), 40(4): 2021b. 20211b.
Yao Feng, Jing Lin, Sai Kumar Dwivedi, Yu Sun, Priyanka Patel, and Michael J Black. Posegpt: Chatting about 3d human pose. arXiv preprint 2rXiv:2311..880 2023.
Patrick Grady, Chengcheng Tang, Christopher D Twigg. Minh Vo, Samarth Brahmbhatt, and Charles C Kemp. Contactopt: Optimizing contact to improve grasps. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1471 1481,, 2021.
Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 3152-5161. 2022.
Yong Guo, Zhiyang Dou, Nan Zhang, Xiyue Liu, Boni Su, Yuguo Li, and Yinping Zhang. Student close contact behavior and covid-19 transmission in china's 's classrooms. PNAS nexus, 2(5): 2023.
Shreyas Hampali, Mahdi Rad, Markus Oberweger, and Vincent Lepetit. Honnotate: method for 3d annotation of hand and object poses. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 3196-3206, 2020.
Xuchen Han, Theodore F. Gast, Qi Guo, Stephanie Wang, Chenfanfu Jiang, and Joseph Teran. A hybrid material point method for frictional contact with diverse materials. 2(2), 2019. doi: 10.1 1411133444225 URL /tnttps://dci.orgggee 0ttp///10.115/33325555
Mohamed Hassan, Vasileios Choutas, Dimitrios Tzionas, and Michael Black. Resolving 3d human pose ambiguities with 3d scene constraints. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 2282-2292, 2019.
Mohamed Hassan, Partha Ghosh, Joachim Tesch, Dimitrios Tzionas, and Michael J Black. Populating 3d scenes by learning human-scene interaction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14708-14718, 2021.
12

---

Published as a conference paper at ICLR 2025
Yana Hasson, Gul Varol, Dimitrios Tzionas, Igor Kalevatykh, Michael Black, Ivan Laptev, and Cordelia Schmid. Learning joint reconstruction of hands and manipulated objects. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 11807-11816, 2019.
Shan He, Haonan He, Shuo Yang, Xiaoyan Wu, Pengcheng Xia, Bing Yin, Cong Liu, Lirong Dai, and Chang Xu. Speech4mesh: Speech-assisted assisted monocular 3d facial reconstruction for speech-driven 3d facial animation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 14192-14202, 2023.
Liwen Hu, Shunsuke Saito, Lingyu Wei, Koki Nagano, Jaewoo Seo, Jens Fursund, Iman Sadeghi, Carrie Sun, Yen-Chun Chen, and Hao Li. Avatar digitization from single image for real-time rendering. ACM Transactions on Graphics (ToG), 1616):1-14,2011
Yuanming Hu, Yu Fang, Ziheng Ge, Ziyin Qu, Yixin Zhu, Andre Pradhana, and Chenfanfu Jiang. A moving least squares material point method with displacement discontinuity and two-way rigid body coupling. ACM Transactions Graphics (TOG), 37(4):1-14, 2018.
Yuanming Hu, Tzu-Mao Li, Luke Anderson, Jonathan Ragan-Kelley, and Frédo Durand. Taichi: language for bigh-performance computation on spatially sparse data structures. ACM Transactions on Graphics (TOG), 38(6):201, 2019.
Buzhen Huang, Liang Pan, Yuan Yang, Jingyi Ju, and Yangang Wang. Neural mocon: Neural motion control for physically plausible human motion capture. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 6417-6426, 2022a.
Chun-Hao Huang, Hongwei Yi, Markus Höschle, Matvey Safroshkin, Tsvetelina Alexiadis, Senya Polikovsky, Daniel Scharstein, and Michael Black. Capturing and inferring dense full-body human-scene contact. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13274-13285, 2022b.
Kemeng Huang, Floyd M. Chitalu, Huancheng Lin, and Taku Komura. Gipe: Fast and stable gauss- newton optimization of ipc barrier energy. 43(2), 2024. ISSN 0730-0301. doi: 10.1 145/3643028. URL /tkp:ttts/////..ogg//// wtw.o////...111134441.11.1
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448 456.. pmlr, 2015.
Angjoo Kanazawa, Michael Black, David W Jacobs, and Jitendra Malik. End-to-end recovery of human shape and pose. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7122-7131, 2018.
Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Metzger, Rodrigo Caye Daudt, and Konrad Schindler. Repurposing diffusion-basse image generators for monocular depth estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Muhammed Kocabas, Chun-Hao Huang, Joachim Tesch, Lea Müller, Otmar Hilliges, and Michael J Black. Spec: Seeing people in the wild with an estimated camera. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 11005-11000 1045, 2021.
Nikos Kolotouros, Georgios Pavlakos, Michael Black, and Kostas Daniilidis. Learning recon- struct 3d human pose and shape via model-fitting in the loop. In Proceedings of the IEEF/CVF international conference on computer vision, pp. 2252-2261, 2019a.
Nikos Kolotouros, Georgios Pavlakos, and Kostas Daniliidis. Convolutional mesh regression for single-image human shape reconstruction. In Proceedings of the IEEF/CVF conference on com- puter vision and pattern recognition, pp. 4501-4510, 2019h.
Raja Kumar, Jiahao Luo, Alex Pang, and James Davis. Disjoint pose and shape for 3d face recon- struction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3115-3125, 2023.
13

---

Published as a conference paper at ICLR 2025
Christoph Lassner, Javier Romero, Martin Kiefel, Federica Bogo, Michael Black, and Peter V Gehler. Unite the people: Closing the loop between 3d and 2d human representations. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 6050-6055 2017.
Chunlu Li, Andreas Morel-Forster, Thomas Vetter, Bernhard Egger, and Adam Kortylewski. Robust model-based face reconstruction through weakly-supervisee outlier segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 372-381, 2023a.
Jiefeng Li, Chao Xu, Zhicun Chen, Siyuan Bian, Lixin Yang. and Cewu Lu. Hybrik: A hybrid analytical-neund inverse kinematics solution for 3d human pose and shape estimation. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3383-3393, 2021.
Jiefeng Li, Siyuan Bian, Qi Liu, Jiasheng Tang, Fan Wang, and Cewu Lu. Niki: Neural inverse kinematics with invertible neural networks for human pose and shape estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12933-12942, 2023b.
Jiefeng Li, Siyuan Bian, Chao Xu, Zhicun Chen, Lixin Yang, and Cewu Lu. Hybrik-x: Hy- brid analytical neurrl inverse kinematics for whole-body mesh recovery. arXiv preprint arXiv:2304. 05690, 2023c.
Kailin Li, Lixin Yang, Haoyu Zhen, Zenan Lin, Xinyu Zhan, Licheng Zhong, Jian Xu, Kejian Wu, and Cewu Lu. Chord: -ategory-level hand-held object reconstruction via shape deformation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9444-9454, 2023d.
Mengcheng Li, Liang An, Hongwen Zhang, Lianpeng Wu, Feng Chen, Tao Yu, and Yebin Liu. Interacting attention graph for single image two-hand reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2761-2770, 2022a.
Minchen Li, Zachary Ferguson, Teseo Schneider, Timothy Langlois, Denis Zorin, Daniele Panozzo, Chenfanfu Jiang, and Danny M. Kaufman. Incremental potential contact: intersection-ant inversion-free, large-deformation dynamics. 39(4), 2020. ISSN 0730-0301. doi: 10.1
$386569.3392425. URL
0ttrg//1..11/33886555 3392425
Tianye Li, Timo Bolkart, Michael J Black, Hao Li, and Javier Romero. Learning a model of facial shape and expression from 4d seans. ACM Truns. Graph., (6666)6194-,,, 2017.
Zhi Li, Soshi Shimada, Bernt Schiele, Christian Theobalt, and Vladislav Golyanik. Mocapdeform: Monocular 3d human motion capture deformable scenes. 2022 International Conference on 3D Vision (3DV), pp. 1-11. IEEE, 2022b.
Zhihao Li, Jianzhuang Liu, Zhensong Zhang, Songcen Xu, and Youliang Yan. Cliff: Carrying location information in full frames into human pose and shape estimation. In European Conference on Computer Vision, pp. 590-606. Springer. 2022c.
Huancheng Lin, Floyd M. Chitalu, and Taku Komura. Isotropic arap energy using cauchy-green invariants. 46)6), 2022. ISSN 0730-0301. doi: 10.1 10.1145/35504443555077 URL https //doi 11511311111/3555550151 35555507..5557
Jing Lin, Ailing Zeng, Haoqian Wang. Lei Zhang, and Yu Li. One-stage 3d whole-body mesh recovery with component aware transformer. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pallern Recognition, pp. 21159-21 168, 2023.
Jing Lin, Ailing Zeng. Shunlin Lu, Yuanhao Cai, Ruimao Zhang, Haoqian Wang, and Lei Zhang, Motion-x: A large-scale 3d expressive woole-body human motion dataset. Advances in Neural Information Processing Systems, 36, 2024.
Kevin Lin, Lijuan Wang, and Zicheng Liu. End-to-end human pose and mesh reconstruction with transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1954-1963, 2021a.
14

---

Published as a conference paper at ICLR 2025
Kevin Lin, Lijuan Wang, and Zicheng Liu. Mesh graphormer. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 12939, 2021b.
Shaowei Liu, Hanwen Jiang, Jiarui Xu, Sifei Liu, and Xiaolong Wang. Semi-supervised 3d hand- object poses estimation with interactions time. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14687-14697, 2021.
Xiyue Liu, Zhiyang Dou, Lei Wang, Boni Su, Tianyi Jin, Yong Guo, Jianjian Wei, and Nan Zhang. Close contact behavior-based covid-19 transmission and interventions in subway system. Journal of Hazardous Materials, 166:12923,, 2022.
Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael Black. Smpl: A skinned multi-person linear model. In Seminal Graphics Papers: Pushing the Boundaries, Volume 2, pp. 1316-66.222 2023.
I Loshchilov. Decoupled weight decay regularization. arXiv preprint arXiv:1711.0510. 2017.
Camillo Lugaresi, Jiuqiang Tang, Hadon Nash, Chris McClanahan, Esha Uboweja, Michael Hays, Fan Zhang, Chuo-Ling Chang, Ming Yong, Juhyun Lee, et Mediapipe: A framework for perceiving and processing reality. In Third workshop computer vision for AR/VR at IEEE computer vision and pattern recognition (CVPR), volume 2019,2019,
Zhengyi Luo, Shun Iwase, Ye Yuan, and Kris Kitani. Embodied scene-aware human pose estimation. Advances in Neural Information Processing Systems. 35:6815-6825 2022.
Gyeongsik Moon. Bringing inputs to shared domains for 3d interacting hands recovery in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17028-17037, 2023.
Gyeongsik Moon and Kyoung Mu Lee. 121-meshnet: Image-to-lixel prediction network for accurate 3d human pose and mesh estimation from a single rgb image. In European Conference Computer Vision, pp. 752-768. Springer, 2020.
Gyeongsik Moon, Shoou-T Yu, He Wen, Takaaki Shiratori, and Kyoung Mu Lee. Interhand2. 6m: A dataset and bascline for 3d interacting hand pose estimation from a single rgb image. In Computer Vision-ECCV ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings. Part XX 16, pp. 548-564. Springer, 2020.
Gyeongsik Moon, Shunsuke Saito, Weipeng Xu, Rohan Joshi, Julia Buffalini, Harley Bellan, Nicholas Rosen, Jesse Richardson, Mallorie Mize, Philippe De Bree, et al. A dataset of relighted 3d interacting hands. Advances in Neural Information Processing Systems, 36, 2024.
Franziska Mueller, Micah Davis, Florian Bernard, Oleksandr Sotnychenko, Mickeal Verschoor, Miguel A Otaduy, Dan Casas, and Christian Theobalt. Real-time pose and shape reconstruction of two interacting hands with a single depth camera. ACM Transactions on Graphics (ToG), 38(4): 1-13, 2019.
Lea Muller, Ahmed AA Osman, Siyu Tang, Chun-Hau Huang, and Michael Black. On self-contact and human pose. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9990-9999, 2021.
Yeonguk Oh, JoonKyu Park, Jaeha Kim, Gyeongsik Moon, and Kyoung Mu Lee. Recovering 3d hand mesh sequence from a single blurry image: A new dataset and temporal unfolding. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 554-563, 2023.
lasonas Oikonomidis, Nikolaos Kyriazis, and Antonis A Argyros. Tracking the articulated motion of two strongly interacting hands. In 2012 IEEE conference computer vision and pattern recognition, pp. 1862-1869. IEEE, 2012.
Christopher Otto, Prashanth Chandran, Gaspard Zoss, Markus Gross, Paulo Gotardo, and Derek Bradley. A perceptual shape loss for monocular 3d face reconstruction. In Computer Graphics Forum, volume 42, pp. e14945. Wiley Online Library, 2023.
15

---

Published as a conference paper at ICLR 2025
Dongwei Pan, Long Zhuo, Jingtan Piao, Huiwen Luo, Wei Cheng, Yuxin Wang, Siming Fan, Shengqi Liu, Lei Yang, Bo Dai, Ziwei Liu, Chen Change Loy, Chen Qian, Wayne Wu, Dahua Lin, and Kwan- Yee Lin. Renderme-360: Large digital asset library and benchmark towards high-fidelity head avatars. In Teirrty-eventhh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023a.
Liang Pan, Jingbo Wang, Buzhen Huang. Junyu Zhang, Haofan Wang, Xu Tang, and Yangang Wang. Synthesizing physically plausible human motions in 3d scenes. arXiv preprint vrXiv:2308.09030 2023b.
JoonKyu Park, Yeonguk Oh, Gycongsik Moon, Hongsuk Choi, and Kyoung Mu Lee. Handocenet: Occlusion-robus 3d hand mesh estimation network. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1496-1505, 2022.
Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed AA Osman, Dimitrios Tzionas, and Michael J Black. Expressive body capture: 3d hands, face, and body from a single image. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10975-10985, 2019.
Xue Bin Peng, Ze Ma. Pieter Abbeel, Sergey Levine, and Angjoo Kanazawa. Amp: Adversarial motion priors for stylized physics-based character control. ACM Transactions Graphics (ToG), 40(4):1-20, 2021.
Xue Bin Peng, Yunrong Guo, Lina Halper, Sergey Levine, and Sanja Fidler. Ase: Large-scale reusable adversarial skill embeddings for physically simulated characters. ACM Transactions On Graphics (TOG), 41(4):1-17, 2022.
Tu-Hoa Pham, Nikolaos Kyriazis, Antonis Argyros, and Abderrahmane Kheddar. Hand-object contact force estimation from markerless visual tracking. IEEE transactions on pattern analysis and machine intelligence, 40(12):2883-2896 2017.
Chandradeep Pokhariya, Ishaan N Shah, Angela Xing, Zekun Li, Kefan Chen, Avinash Sharma, and Srinath Sridhar. Manus: Markerless hand-object grasp capture using articulated 3d gaussians. arXiv preprint arXiv:2312.02137 2023.
Albert Pumarola, Antonio Agudo, Aleix M Martinez, Alberto Sanfeliu, and Francese Moreno-Noo -Nguur. Ganimation: Anatomically-awart aware facial animation from single image. In Proceedings of the Eurupean conference on computer vision (ECCV), pp. 818-833, 2018
Dafei Qin, Jun Saito, Noam Aigerman, Thibault Groueix, and Taku Komura. Neural face rigging for animating and retargeting facial meshes in the wild. In ACM SIGGRAPH 2023 Conference Proceedings, pp. 1-11, 2023.
Nikhila Ravi, Jeremy Reizenstein, David Novotny, Taylor Gordon, Wan-Yen Yen Lo, Justin Johnson, and Georgia Gkioxari. Accelerating 3d deep learning with pytorch3d. arXiv preprint arXiv:2007.08501, 2020.
Davis Rempe, Leonidas J Guibas, Aaron Hertzmann, Bryan Russell, Ruben Villegas, and Jimei Yang. Contact and human dynamics from monocular video. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part V 16, pp. 71-87. Springer, 2020.
Davis Rempe, Tolga Birdal, Aaron Hertzmann, Jimei Yang, Srinath Sridhar, and Leonidas J. Guibas. Humor: 3d human motion model for robust pose estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 11488-11490 11499, October 2021.
Javier Romero, Dimitrios Tzionas, and Michael J Black. Embodied hands: Modeling and capturing hands and bodies together. arXiv preprint mrXii:2201.026/10 2022.
Yu Rong. Jingbo Wang, Ziwei Liu, and Chen Change Loy. Monocular 3d reconstruction of interacting hands via collision-ans aware factorized refinements. In 2021 International Conference on 3D Vision (3DV), pp. 432-441. IEFE, 2021.
16

---

Published as a conference paper at ICLR 2025
Soubhik Sanyal, Timo Bolkart, Haiwen Feng. and Michael Black. Learning to regress 3d face shape and expression from an image without 3d supervision. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 7763-7772, 2019.
Mingyi Shi, Sebastian Starke, Yuting Ye, Taku Komura, and Jungdam Won. Phasemp: Robust 3d pose estimation via phase-conditioned human motion prior. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 14725-14737, 2023.
Soshi Shimada, Vladislav Golyanik, Zhi Li, Patrick Pérez, Weipeng Xu, and Christian Theobalt. Hule: 3d human motion capture with pose manifold sampling and dense contact guidance. In European Conference on Computer Vision, pp. 516-533. Springer, 2022.
Soshi Shimada, Vladislav Golyanik, Patrick Pérez, and Christian Theobalt. Decaf: Monocular deformation capture for face and hand interactions. ACM Transactions on Graphics (TOG), 42(6): 1-16, 2023.
Jan L. Spille, Martin Grunwald, Sören Martin, and Stefanie M Mueller. Stop touching your face! systematic review of triggers, characteristics, regulatory functions and neuro-physiology of facial self-touch. Neuroscience & Biobehavioral Reviews, 128:102-116, Sep 2021. doi: 10.1016/j. neubiorev.0021.05.055
Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang. Deep high-resolution representation learning for human pose estimation. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pp. 5693-5703, 2019.
Bugra Tekin, Federica Bogo, and Marc Pollefeys. H+ o: Unified egocentric recognition of 3d hand-object poses and interactions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 451 4511-4520, 2019.
Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen-Or, and Amit H Bermano. Human motion diffusion model. arXiv preprint arXiv:2209. 14916, 2022.
Justus Thies. Michael Zollhofer. Mare Stamminger, Christian Theobalt, and Matthias Nießner. Face2face: Real-time face capture and reenactment of rgb videos. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2387-2395, 2016.
Edith Tretschk, Navami Kairanda, Mallikarjun BR, Rishabh Dabral, Adam Kortylewski, Bernhard Egger, Mare Habermann, Pascal Fua, Christian Theobalt, and Vladislav Golyanik. State of the art in dense monocular non-rigid 3d reconstruction. In Computer Graphics Forum, volume 42, pp. 485-520. Wiley Online Library, 2023.
Aggeliki Tsoli and Antonis A Argyros. Joint 3d tracking of a deformable object in interaction with hand. Proceedings of the European Conference on Computer Vision (ECCV), pp. 484-500, 2018.
Weilin Wan, Zhiyang Dou, Taku Komura, Wenping Wang, Dinesh Jayaraman, and Lingjie Liu. Tlcon- trol: Trajectory and language control for human motion synthesis. arXiv preprint arXiv:2311.17135, 2023a.
Weilin Wan, Yiming Huang, Shutong Wu, Taku Komura, Wenping Wang, Dinesh Jayaraman, and Lingjie Liu. Diffusionphase: Motion diffusion in frequency domain. arXiv preprint TrXiv:2312.04036. 2023b.
Jiayi Wang. Franziska Mueller, Florian Bernard, Suzanne Sorli, Oleksandr Sotnychenko, Neng Qian, Miguel A Otaduy, Dan Casas, and Christian Theobalt. Rgb2hands: real-time tracking of 3d hand interactions from monocular rgb video. ACM Transactions Graphics (ToG), 39(6):1-16, 2020.
Jingbo Wang, Yu Rong, Jingyuan Liu, Sijie Yan, Dahua Lin, and Bo Dai. Towards diverse and natural scene-aware human motion synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20469, 2022a.
Lizhen Wang, Zhiyuan Chen, Tao Yu, Chenguang Ma, Liang Li. and Yebin Liu. Faceverse: fine- grained and controllale 3d face morphable model from hybrid dataset. In Proceedings of the IEFF/CVF conference computer vision and pattern recognition, pp. 20333-20342. 2022b.
17

---

Published as a conference paper at ICLR 2025
Wenjia Wang, Yongtao Ge, Haiyi Mei, Zhongang Cai, Qingping Sun, Yanjun Wang, Chunhua Shen, Lei Yang, and Taku Komura. Zolly: Zoom focal length correctly for perspective- human mesh reconstruction. arXiv preprint arXiv:2303. 2023a.
Yanjun Wang, Qingping Sun, Wenjia Wang, Jun Ling, Zhongang Cai, Rong Xie, and Li Song. Learning dense uv completion for human mesh recovery. arXiv preprint arXiv:2307.11074 2023b.
Yufu Wang and Kostas Daniilidis. Refit: Recurrent fitting network for 3d human recovery. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 14644-14654, 2023.
Shih-En Wei, Jason Saragih, Tomas Simon, Adam W Harley, Stephen Lombardi, Michal Perdoch, Alexander Hypes, Dawei Wang, Hernan Badino, and Yaser Sheikh. Vr facial animation via multiview image translation. ACM Transactions on Graphics (TOG), 38(4):1-16, 2019.
Zhenzhen Weng and Serena Yeung. Holistic 3d human and scene mesh estimation from single view images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 334-343, 2021.
Erroll Wood, Tadas Baltrušaitis, Charlie Hewitt, Matthew Johnson, Jingjing Shen, Nikola Milosavlje- vić, Daniel Wilde, Stephan Garbin, Toby Sharp, Ivan Stojiljković, et al. 3d face reconstruction with dense landmarks. European Conference on Computer Vision pp. 160-177. Springer, 2022.
Xianghui Xie, Bharat Lal Bhatnagar, and Gerard Pons-Moll. Chore: Contact, human and object reconstruction from a single rgb image. European Conference on Computer Vision, pp. 125-145. Springer, 2022.
Sirui Xu, Zhengyuan Li, Yu-Xiong Wang, and Liang- Yan Gui. Interdiff: Generating 3d human-object interactions with physics-informed diffusion. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 14928-14940, 2023.
Sirui Xu, Ziyin Wang, Yu-Xiong Wang, and Liang- Yan Gui. Interdreamer: Zero-shot text to 3d dynamic human-object interaction. In NeurIPS, 2024.
Lixin Yang, Xinyu Zhan, Kailin Li, Wenqiang Xu, Jiefeng Li, and Cewu Lu. Cpf: Learning a contact potential field to model the hand-object interaction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1099-111066 2021.
Lixin Yang, Kailin Li, Xinyu Zhan, Jun Lv, Wenqiang Xu, Jiefeng Li, and Cewu Lu. Artiboost: Boosting articulated 3d hand-object pose estimation via online exploration and synthesis. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pp- 2750-2760, 2022a.
Lixin Yang, Kailin Li, Xinyu Zhan, Fei Wu, Anran Xu, Liu Liu, and Cewu Lu. Oakink: A large-scale knowledge repository for understanding hand-object interaction. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, 20953-20962, 2022b.
Xueze Yang, Zhiyang Dou, Yuqing Ding, Boni Su, Hua Qian, and Nan Zhang. Analysis of sars- cov-2 transmission in airports based on real human close contact behaviors. Journal of Building Engineering, 82: 10829, 2024.
Yufei Ye, Abhinav Gupta, and Shubham Tulsiani. What's in your hands? 3d reconstruction of generic objects in hands. In Proceedings of the IEEESCVF Conference on Computer Vision and Pattern Recognition, pp. 3895-3905, 2022.
Yufei Ye, Poorvi Hebbar, Abhinav Gupta, and Shubham Tulsiani. Diffusion-guiie guided reconstruction of everyday hand-object interaction clips. In ICCV, 2023a.
Yufei Ye, Xueting Li, Abhinav Gupta, Shalini De Mello, Stan Birchfield, Jiaming Song, Shubham Tulsiani, and Sifei Liu. Affordance diffusion: Synthesizing hand-object interactions. CVPR, 2023b.
18

---

Published as a conference paper at ICLR 2025
Zhengdi Yu, Shaoli Huang, Chen Fang. Toby P Breckon, and Jue Wang. Acr: Attention collaboration- based regressor for arbitrary two-hand reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12955-12964, 2023.
Baowen Zhang, Yangang Wang, Xiaoming Deng, Yinda Zhang, Ping Tan, Cuixia Ma, and Hongan Wang. Interacting two-hand 3d pose and shape reconstruction from single color image. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1 11354-11365 2021a.
Hongwen Zhang, Yating Tian, Xinchi Zhou, Wanli Ouyang, Yebin Liu, Limin Wang, and Zhenan Sun. Pymaf: 3d human pose and shape regression with pyramidal mesh alignment feedback loop. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 11446-11456, 2021b.
Jason Y Zhang, Sam Pepose, Hanbyul Joo, Deva Ramanan, Jitendra Malik, and Angjoo Kanazawa. Perceiving 3d human-object spatial arrangements from a single image in the wild. In Computer Aision-ECCI 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Purt XII 16, pp. 34-51. Springer, 2020a.
Nan Zhang. Li Liu, Zhiyang Dou, Xiyue Liu, Xueze Yang, Doudou Miao, Yong Guo, Silan Gu, Yuguo Li, Hua Qian, et al. Close contact behaviors of university and school students in 10 indoor environments. Journal of Hazardous Materials, 458:132069, 2023a.
Nan Zhang, Xiyue Liu, Shuyi Gao, Boni Su, and Zhiyang Dou. Popularization of high-speed railway reduces the infection risk via close contact route during journey. Sustainable Cities and Sociely, 99:104979, 2023b.
Nan Zhang, Xueze Yang. Boni Su, and Zhiyang Dou. Analysis of sars-cov-2 transmission in univer- sity classroom based on real human close contact behaviors. Science of The Total Environment, 917:170346, 2024.
Siwei Zhang, Yan Zhang, Federica Bogo, Mare Pollefeys, and Siyu Tang, Learning motion priors for 4d human body capture in 3d scenes. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1 11343-1 1353, 2021c.
Tianke Zhang, Xuangeng Chu, Yunfei Liu, Lijian Lin, Zhendong Yang. Zhengzhuo Xu, Chengkun Cao, Fei Yu, Changyin Zhou, Chun Yuan, et al. Accurate 3d face reconstruction with facial component tokens. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9033-9042, 2023c.
Yan Zhang, Mohamed Hassan, Heiko Neumann, Michael Black, and Siyu Tang. Generating 3d people in scenes without people. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 6194-6204, 2020b.
Qingcheng Zhao, Pengyu Long, Qixuan Zhang, Dafei Qin, Han Liang, Longwen Zhang, Yingliang Zhang, Jingyi Yu, and Lan Xu. Media2face: Co-speech facial animation generation with multi- modality guidance. arXiv preprint arXiv:2401.15687, 2024.
Yufeng Zheng, Victoria Fernández Abrevaya, Marcel Bühler, Xu Chen, Michael Black, and Otmar Hilliges. Im avatar: Implicit morphable head avatars from videos. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 13545-13555, 2022.
Wenyang Zhou, Zhiyang Dou, Zeyu Cao, Zhouyingcheng Liao, Jingho Wang, Wenjia Wang, Yuan Liu, Taku Komura, Wenping Wang, and Lingji Liu. Emdm: Efficient motion diffusson model for fast, high-quality motion generation. arXiv preprint arXiv:2312.02233 2023.
Wojciech Zielonka, Timo Bolkart, and Justus Thies. Towards metrical reconstruction of human faces. In European Conference Computer Vision. pp. 259-269. Springer, 2022.
Christian Zimmermann, Duygu Ceylan, Jimei Yang, Bryan Russell, Max Argus, and Thomas Brox. Freihand: A dataset for markerless capture of hand pose and shape from single rgb images. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 313-822,73 2019.
Binghui Zuo, Zimeng Zhao, Wenqian Sun, Wei Xie, Zhou Xue, and Yangang Wang. Reconstructing interacting hands with interaction prior from monocular images. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 9054-9064, 2023.
19

---

Published as a conference paper at ICLR 2025
Keypoints and Vertices [xx44]
Deformation [xya] and Contacts [o] (4-Dim)
[3 BB-im)
Transformer Encoder (IK/8)-Dim)
Transformer Encoder (HHS))-Dim)
MLP
Norm
Transformer Encoder (1H/4) Dim)
Transformer Encoder
((H44)Dim))
Multi-Head Salf Attention
Transformer Encoder (H/2)
Fully Connected
Norm
Image Features
Image [H++] Dim
(H-3)-Dim
(a)
Figure 5: Structural details of the MeshNet and InteractionNet. (a) MeshNet; (b) InteractionNet; (c) Internal structure of a Transformer Encoder block.
A IMPLEMENTATION DETAILS
A. CNN BACKBONE
The CNN backbone used in our framework is an HRNet-W64 (Sun et al. 2019). initialized with imagenet-pretrained weights. The weights of the backbone would be updated during training. We extract a (49 x H)-dim feature map from this network and upsamples it to a (N x H)-dim feature map, where N No. + Ns,, + No. + Nh.. the total number of head and hand keypoints Nb. N and vertices N,., ,N.. Then, we concatenate the keypoints and the vertices corresponding to the head and hand mean pose as keypoints and vertex queries, resulting map. Random masking of keypoints and vertex queries of rate 30% is applied, following (Lin et al. 2021a).
((N + 3) x H)-dim feature
A.2 MESHNET AND INTERACTIONNER
Our MeshNet and InteractionNet have similar progressive downsampling transformer encoder struc- tures, Fig. for illustration. The MeshNet has three component transformer encoders with decreasing feature dimensions. The InteractionNet starts with a fully connected layer that downsam- ples the feature dimension, followed by two transformer encoders. Each transformer encoder has Multi-Head Attention module consisting of 4 layers and 4 attention heads. In addition to head and hand mesh features, MeshNet also regresses head and hand keypoints, which are only for supervision and not used by any downstream components.
A.3 IKNET
Our IKNets take in rough mesh features V V and output the pose and shape parameters (0, B). as well as the global rotation and translation RR T). They feature a Multi-Layer Perceptron (MLP)
structure, each consisting of five MLP Blocks and final fully connected layer. Each MLP Block contains a fully connected layer, followed by a batch normalization layer (Ioffe & Szegedy 2015) and ReLU activation layer. There are two skip-connections, connecting the output of the first block with the input of the third block, and the output of the third block with the input of the final fully connected layer. See Fig. for an illustration. The hand and head IKNets have the same structure, differing only in their input and output dimensions. The hidden dimensions the two IKNets are 1024.
20

---

Published as a conference paper at ICLR 2025
R.7.
FuByy
ML Bick
MLD Block
MLP Block
ML7 Block
Biok
Mech Features
Figure 6: Structural details of the IKNet.
Input
Without Depth
With Depth
Figure 7: Qualitative demonstration of the effects of the depth loss. The model generalizes poorly in the 7-direction when trained without depth supervision.
A.4 TRAINING AND TESTING DETAILS
To be consistent with the training setting of Decaf (Shimaaa et al. 2023), in the Decaf dataset, we all eight camera views and the subjects S2, S4, S5, S7, and S8 in the training data split for training. For testing, we use only the front view (view 108) and the subjects S1, S3, and S6 the testing data split. The low, mid, and high-resolution head mesh consists of 559, 1675, and 5023 vertices, respectively. The low and high-resolution hand mesh consists of 195 and 778 vertices, respectively. We the middle-resolution head mesh and the high-resolution hand mesh the inputs of head and hand IKNets.
B MORE QUALITATIVE COMPARISONS
We demonstrate qualitatively the effect of the absence of the depth loss Fig. 7 When trained without depth loss, the network only supervised with 2D information in-the-wild data, without any constraints the z-direction. As a result, artifacts such as self-penetration frequently oceur in this case. The introduction of depth loss eliminates this ambiguity, allowing the correct relative positioning of hand and face.
CConfirmed by the authors of Decaf
21

---

Published as a conference paper at ICLR 2025
C ADDITION DETAILS ON LOSSES
Here, we provide the details of the mesh losses and the interaction losses. The details of the adversarial loss and the depth loss are already mentioned in the main paper.
C.I
MESH LOSSES
The mesh loss mesh consists of four components.
Lessh
Lreproj 4.ven + 2Lexy + 2a
(6)
Vertices Loss. L1 loss is used for predicted rough 3D face and hand vertices V/, V FLAME- regressed undeformed 3D face vertices Vj and MANO-regressed 3D hand vertices VA against the ground-truth 3D undeformed face vertices V and 3D hand vertices Vn-
Lvert An (Mnnoam
|| V% Va +IV, V Ay (Hmonparr I Vxlli+IVV; V (7)
where A5 empirically set to 3 and 1 respectively. Mnnonpara is set to to emphasize the supervision on the more complex non-parametric mesh features.
Keypoints Loss. We use L1 loss for predicted rough 3D face and hand keypoints K'J K, 3D face and hand keypoints extracted from rough mesh K fraas Kr FLAMB-regressed 3D face keypoints K and MANO-regressed 3D hand keypoints K₁, against the ground-truth 3D undeformed face keypoints K, and 3D hand keypoints Ky
Layy = Pnomparal (1
K Killi + Khrac
KAlli + |K' K + |K K
(8)
+ || R [K, K
Where Macopara empirically set 4, put more weight on the non-parametric mesh with high degrees of freedom.
Reprojection loss. L, loss used for reprojected rough 3D face and hand keypoints Kj, K2,- 3D face and hand keypoints extracted from rough mesh Kusat K K FLAMEE-rgressedd 3D face keypoints K, and MANO-regressed 3D hand keypoints K,, against the ground-trum -ound-truth face and hand 2D keypoints K K King
Creproj (III(())) K Ao 1l1 + (II( (ima)) Ktoll 11 (I[[K,),
K l12)
(10)
+XX(IJU(()))) - R20 11 |II[K Frcis) (ucas) K f20 + [II(K,)) K /zm Il1)
(11)
Where II is the learned camera projection function. An. A are set to 4 and 1 respectively.
Parameter loss. apply L1 loss on the regressed hand and face pose, shape, and facial expression parameters against their respective ground truths.
Crace-params = (1.3r - Rdll1 + |10 01ecpp12 1 + -101=pose 0p-posell 1113
(12)
Lhand-params (1.3 Bhlli 110 0nll
(13)
Gparams Lisce-params Lhaad--pramss
(14)
C.2 INTERACTION LOSSES.
The interaction loss Lintercctonn consists of four components.
Cinteraction = 0.2Loouch + 0.6L Lcollision + 6.
(15)
Deformation loss. Due to the human anatomy, some vertices on the face are more easily deformed than other vertices. Therefore, we impose adaptive weighting each vertex, and square
22

---

Published as a conference paper at ICLR 2025
(a)
(b)
(c)
(d)
Figure 8: Examples of failed keypoint estimation in case of large self-occlusion. (a) input image; (b) inaccurate keypoint estimation by the same keypoint estimators used in Decaf (Lugaresi et al. 2019 Bulat & Tzimiropoulos/ 2017); (c) reconstructed hand-fac face interaction by method. (d) reconstructed hand face interaction Decaf.
loss to penalize large deformation. We also have regularization term to penalize extremely large deformations.
Ldeform
(ld+lalla))dd 4
d IEC
(16)
EET
Where is the set of indices of face vertices, d,, de are the predicted and ground truth deformation vector for index i, and C (i I: ||d/ 3cm) the vertices of large deformations. fa and X are empirically set to be 5000, 100 respectively.
Touch loss. Let V Fc and V He denote the set of face and hand vertices that are predicted by the model to have contact probability greater than 0.5.
Luunh CD(V Pe: VHa) + CD(V He: Vra)
(17)
Where CD(X, Y) gives the mean Chamfer Distance (CD) between each point in X to the closest point in Y.
Collision loss. Let V Hoa denote the set of hand vertices that penetrates the face surface, VF and Dr denote the predicted face mesh vertices and deformations.
Leallision
CD(V Hox
VF -DF)
(18)
Contact loss. Let CH and Cp denote the predicted hand and face contact probabilities and Č₁. Ör denote the ground-truth contact labels.
Coontact BCE(CH, Ōτ) + BCE(CP, ÖF)
(19)
Where BCE denote the binary cross-entropy loss.
D MORE DISCUSSIONS
D.1 PERFORMANCE UNDER CHALLENGING OCCLUSION.
As seen in Fig. our end-to-end DICE method is robust under challenging self-occlusion cases, such as the hand covering more than half of the face. On the other hand, Decaf (Shimada al.) 2023), which requires an initial keypoint prediction for test-time optimization, performs poorly in this situation.
23

---

Published as a conference paper at ICLR 2025
(b)
Input Image
Output Mesh
Input Image
Output Mesh
Figure 9: Examples of failure cases in case of complete occlusion of the hand. (a) Hand or face completely occluded. (b) Out-of-distriuttion data.
D.2 FAILURE CASES
In Fig. 9 demonstrate the failure cases of method. When the hand extremely far from the face, or when the hand completely obscured by the head, our method could fail to reconstruct the hand-face interaction. Also, when given out-o--distributtin data, such as when the hand is wearing gloves or the input subject is an infant, the reconstruction accuracy could degrade.
D.3 SOCIETAL IMPACT
D.3.1 POTENTIAL MISUSE
DICE enables tracking of individuals' appearances, gestures, and interactions with high fidelity, there is risk that it may be misused for negative applications, such as surveillance, and may cause privacy infringement. Also, since DICE makes use of a readily animatable representation, it could enable realistic deepfakes driven by the pose and shape information collected, which could be used in creating misinformation and conducting identity theft. We are firmly against any form of misuse of the DICE model.
D.3.2 DATA FAIRNESS
As hand-face interaction recovery is a human-related task, data fairness is critical. The currently used Decaf [Shiadaa et al. (2023) dataset needs improvement in the inclusion of human actors from underrepresented demographic groups. This may result in a model trained only on Decaf underperforming input data on such groups, perpetuating inequality and limiting equitable access. Our weak- supervise training scheme introduces diverse in-the-wild data, which could alleviate this issue as the amount of wild data scales up.
24


Reviews:
[{"review_id": "Q0ODZbks4m", "reviewer": "Reviewer_6ptN", "summary": "This paper presents a model for the simultaneous reconstruction of face and hand meshes from a single image, based on previous work and annotated data from Shimada et al. (DECAF). The model employs a Transformer-based image encoder to extract keypoints for face and hand through separate streams, utilizing parametric models for reconstruction. Designed for images that contain both a face and a hand, the approach extends training with unannotated data by leveraging a diffusion-based depth estimation model (Ke et al., 2024) to guide mesh reconstruction. After reconstructing the face and hand mesh, the depth maps are computed and aligned with depth estimates from the diffusion model. The model is trained with four types of losses: mesh loss and contact-point loss, similar to those in DECAF, as well as depth loss and adversarial loss, which support the depth supervision process. Experimental results show improvements over previous work for face-hand interaction images.", "questions": "Is it possible to apply depth supervision to other reconstruction tasks? What makes it particularly suited to the hand-face problem, aside from data availability? Could this approach be more broadly relevant to visual tasks involving contact points between two objects? I wonder if this could be explored further.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "-- The use of depth supervision for weakly-supervised training in mesh reconstruction is innovative and potentially impactful for similar tasks.", "weaknesses": "-- The model is focused specifically on face-hand images, which may limit its relevance to the broader computer vision community. The emphasis on this specific interaction appears to be largely application-driven, leaving questions about its applicability to other computer vision tasks.\n\n-- Although the use of depth supervision in training mesh models appears novel, it leverages known components, which slightly reduces the overall technical novelty.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ztnwDCkwMr", "reviewer": "Reviewer_fJBo", "summary": "The paper introduces a novel method for reconstructing 3D hand-face interactions with deformations from a single image. The authors present DICE, a Transformer-based architecture that simultaneously estimates hand and face poses, contacts, and deformations. A key feature of DICE is its ability to disentangle local deformation fields from global mesh vertex locations into two separate network branches, which enhances the accuracy of deformation and contact estimation for precise hand-face mesh recovery. Additionally, the method utilizes a weakly-supervised training approach with in-the-wild images that lack 3D ground-truth annotations, thereby improving the model's generalizability. Experiments show state-of-the-art performance in terms of accuracy and physical plausibility, along with interactive inference rates on modern GPUs.", "questions": "Explain the notable gap in Decaf regarding physics plausibility metrics.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The motivation is compelling. DICE provides the first end-to-end learning-based method for capturing hand-face interactions and deformations from a single image, filling a significant gap in the field.\n* Good experimental results. The paper claims that DICE achieves superior reconstruction quality compared to baseline methods while operating at an interactive rate (20 fps), which is crucial for real-time applications.\n* Technically sound. The weakly-supervised training scheme using in-the-wild images enhances the model's ability to generalize beyond the constraints of studio-collected data.", "weaknesses": "* DICE is better than optimization-based methods like Decaf. However, there is a notable gap in physics plausibility metrics compared to Decaf. Can the authors provide a clearer explanation? I wonder if it struggles to balance reconstruction accuracy and physics plausibility.\n* In Table 3, using in-the-wild data negatively impacts the F-score. The authors should provide a clearer explanation.\n* The novelty is limited. The techniques used in this paper are commonly employed in the community.\n* The authors are encouraged to include more discussion on failure cases.\n* Some results are not convincing. In Figure 4, the head reconstruction of Decaf is noticeably better than that of the authors' method for the third sample from in-the-wild data.\n* Some videos should be included to demonstrate real-time capability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Nh9P0nXlvQ", "reviewer": "Reviewer_mMEY", "summary": "### Motivation\n- Modeling the geometry and interactions of human hands and face from a monocular image is an underexplored tasks in 3DCV, with broad applications (AR/VR, motion capture, etc.).\n- To the best of the authors' knowledge (and mine), only one prior method tackle this problem in an exhaustive and end-to-end manner, Decaf [Shimada et al., 2023]. This seminal work is however optimization-based, and therefore too slow for interactive use.\n\n### Contributions\n- The authors thus introduce DICE, which performs the above-mentioned task in a forward-inference manner. The method relies on a Transformer-based network, with two branches separately tackling the regression of (a) the hand+face geometry and (b) the face deformation caused by hand contact.\n- Furthermore, the authors introduce a weak supervision to expand training to in-the-wild-ish images (2D keypoint annotations are still needed, though not 3D ones). Depth supervision is also enabled on such data by integrating a pre-trained monocular depth estimator to generate pseudo ground-truth. Predicted meshes are projected into depth images using a differentiable renderer for comparison.\n\n\n### Results\n- Experiments on the Decaf benchmark show that DICE outperforms prior art in terms of 3D reconstruction accuracy and provides competitive runtime on par with regression-based baselines (~0.1s, whereas Decaf needs ~15s). Unlike prior art though, the authors did not fully disclose their results on physics-plausibility relating to touchness and collision.\n- An ablation study, as well as plenty of convincing qualitative results, are also provided. \n\n### Relevance\n- This paper addresses a relevant yet under-explored 3DCV task. The community would benefit from this iteration over the seminal Decaf.", "questions": "_see **Weaknesses** for key questions/remarks._\n\n\n### Q1. Hand Rigidity $\\rightarrow$ Applicability of CONTHO\n- One limitation of this method, as well as prior Decaf, is to only consider face deformation and ignore hand one. While the hands are definitely _stiffer_, ignoring their soft tissues can yield incorrect results in some cases (e.g., hand pressed against forehead). Moreover, by considering the hand as piece-wise rigid, the target task becomes similar to the much less under-explored domain of human/object interaction modeling (i.e., both tasks can be expressed as modeling 3D non-rigid/rigid interactions). It seems that recent solutions such as CONTHO [Nam et al., 2024] could be adapted to face/hand reconstruction. What is the authors' position w.r.t. these adjacent methods? How would the authors differentiate their own approach?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "_(somewhat ordered from most to least important)_\n\n### S1. Clear Explanations and Decent Reproducibility\n- The Methodology is clear, thoroughly explained, and overall well-formalized. \n- While the actual implementation has not been provided, an expert in the art should be able to re-implement this work.\n- The paper is also well illustrated and easy to follow (pipeline figures + qualitative results).\n- Limitations are well-discussed, with failure cases provided (Appendix D).\n\n### S3. Convincing Evaluation\n- The evaluation is also clear, comprehensive, and well presented (e.g., highlighting optimization-based vs. regression-based methods in Table 1, providing various qualitative results, etc.).\n- The proposed DICE method outperforms the SOTA on 3D reconstruction metrics, at a competitive running time.\n- Qualitative results and the ablation study are also convincing. In-the-wild examples help illustrate the generalizability of the method.\n\n### S3. End-to-End Solution Targeting an Under-explored Task\n- As mentioned above, this paper is seemingly only the second (after Decaf [Shimada et al., 2023]) to address the joint modeling of hand/face geometry and interaction in a comprehensive, end-to-end manner. As such, it would be valuable to the community.\n- While the proposed method mostly relies on existing components (CNN-backbone, Transformer processing, weak supervision using large monocular depth regressor, etc.), it holds value as a well-designed end-to-end system.", "weaknesses": "_(somewhat ordered from most to least important)_\n\n\n### W1. Limited Technical Novelty\n- The proposed solution has value as an end-to-end system tackling an under-explored task, but its technical novelty is limited.\n- The main/only technical contribution claimed by the authors are their weak depth supervision module, relying on a large/foundation model [L110-112]. There might be some novelty in applying such a model to this specific task (e.g., in terms of processing the depth-map predictions to supervise the facial keypoints), but the underlying idea of leveraging a large monocular depth model for weak supervision is not novel.\n- Most of the loss functions appear borrowed from Decaf [Shimada et al., 2023], even though this is not made obvious in the paper (both in Section 3.3. and in Appendix C).\n\n\n\n### W2. Lack of Insight w.r.t. Physics Metrics\n- Unlike Decaf, the authors here did not report all physics-related metrics (non-collision / touchness accuracy).\n- The authors could provide more insight in terms of the challenges faced by regression-based methods for accurate modeling, compared to optimization-based solutions.\n- Similarly, Table 2 and corresponding paragraph [L417-421] lack proper insight/description. The superiority of DICE is not as obvious as claimed.\n\n### W3. Somewhat Limited Comparison to Prior Art\n- The authors compare their method to 3 other solutions: Decaf [Shimada et al., 2023] (most relevant as it also targets hand/face interaction), PIXIE [Feng et al., 2021a] and METRO [Lin et al., 2021a)] (both originally developed for full-body regression), and a baseline/benchmark solution [Lugaresi et al., 2019; Li et al., 2017)] (composed of two independent hand-only and face-only models). While the literature does indeed lack in prior art focusing on hand/face interaction, more baseline methods could have been considered, e.g., mixing other single-target methods (MinimalHand , RingNet, FastMETRO, etc.). This could have further highlighted the benefits on modeling both body parts jointly.\n- To counterbalance this \"weakness\", it should be noted that the aforementioned set of prior methods have been entirely borrowed from Decaf, so the authors here stuck to Decaf's protocol, which is fair. More comparisons could make the paper stronger.\n\n### W4. Minor Remarks\n- Societal impact is not discussed, showing a lack of self-reflection. While positive applications are actually listed in the Introduction (e.g., benefits to AR/VR and gaming industry), a variety of negative applications can also come to mind: surveillance (appearance + gesture), animatable deepfakes, etc. Discussions on data fairness/bias are also always relevant when it comes to human-modeling solutions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)