DICE: End-to-end Deformation Capture of Hand-Face Interactions from a Single Image

## Abstract

Abstract Reconstructing 3D hand-face interactions with deformations from a single image is a challenging yet crucial task with broad applications in AR, VR, and gaming. The challenges stem from self-occlusions during single-view hand-face interactions, diverse spatial relationships between hands and face, complex deformations, and the ambiguity of the single-view setting. The first and only method for hand-face interaction recovery, Decaf [ 97 ] , introduces a global fitting optimization guided by contact and deformation estimation networks trained on studio-collected data with 3D annotations. However, Decaf suffers from a time-consuming optimization process and limited generalization capability due to its reliance on 3D annotations of hand-face interaction data.
To address these issues, we present DICE , the first end-to-end method for D eformation-aware hand-face I nteraction re C ov E ry from a single image. DICE estimates the poses of hands and faces, contacts, and deformations simultaneously using a Transformer-based architecture. It features disentangling the regression of local deformation fields and global mesh vertex locations into two network branches, enhancing deformation and contact estimation for precise and robust hand-face mesh recovery. To improve generalizability, we propose a weakly-supervised training approach that augments the training set using in-the-wild images without 3D ground-truth annotations, employing the depths of 2D keypoints estimated by off-the-shelf models and adversarial priors of poses for supervision.
Our experiments demonstrate that DICE achieves state-of-the-art performance on a standard benchmark and in-the-wild data in terms of accuracy and physical plausibility. Additionally, our method operates at an interactive rate (20 fps) on an Nvidia 4090 GPU, whereas Decaf requires more than 15 seconds for a single image. Our code will be publicly available upon publication.

## 1 Introduction

Hand-face interaction is a common behavior observed up to 800 times per day across all ages and genders []. Therefore, faithfully recovering hand-face interactions with plausible deformations is an important task given its wide applications in AR/VR [, , ], character animation [, ], and human behavior analysis [, , ]. Given the speed requirement of downstream applications like AR/VR, fast and accurate 3D reconstruction of hand-face interactions is highly desirable.
However, several challenges make monocular hand-face deformation and interaction recovery a cumbersome task: 1) the self-occlusion involved in hand-face interaction, 2) the diversity of hand and face poses, contacts, and deformations, and 3) ambiguity in the single-view setting.
Most existing methods [, ] only reconstruct the hand [] and face [] meshes, or unified as a whole body [, ] with the body parts, without capturing contacts and deformations. A seminal advance, Decaf [], recovers hand-face interactions with deformations and contacts taken into account. However, it requires time-consuming optimization, which takes more than 15 seconds per image, rendering it unsuitable for interactive applications. The iterative fitting process of Decaf relies on an accurate estimation of hand and face keypoints and contacts on the hand and face surfaces, which could fail when significant occlusion is present in the image; See Fig. 8 in the Appendix D.1. Additionally, Decaf cannot scale up their training to fruitful hand-face interaction data in the wild, as they require 3D ground-truth annotations, i.e., contact labels and deformations.

Figure: Figure 1: Our method, DICE, is the first end-to-end approach that captures hand-face interaction and deformation from a monocular image. (a) Decaf validation dataset. (b) In-the-wild images. (c) Use-cases in VR.

To tackle the issues above, we present DICE, the first end-to-end approach for Deformation-aware hand-face Interaction reCovEry from a monocular image. We use a Transformer-based model with the attention mechanism to effectively capture hand-face relationships.
Motivated by the global nature of the pose and shape of hand and face and the local nature of the deformation field and contact probabilities, we further propose disentangling the regression of deformation from the pose and shape of hand and face represented by mesh vertex positions into two network branches, which enhances the estimation of deformations and contacts while resulting in accurate and robust hand and face mesh recovery. Instead of directly regressing hand and face parameters, we learn an intermediate non-parametric mesh representation.
We then use this representation to regress the pose and shape parameters of hand and face with a neural inverse-kinematics network. Compared with directly regressing the pose and shape parameters which learns the abstract parameters is a highly non-linear process and suffers from image-model misalignment, predicting vertex positions in Euclidean space and then applying inverse-kinematics enhances the reconstruction accuracy [, , ].
Consequently, our model achieves higher reconstruction accuracy than all previous regression and optimization-based methods. It also reaps the benefits of an animatable parametric hand and face representation that could be readily used by downstream applications.

Meanwhile, despite containing rich annotations, the existing benchmark dataset [] collected in a studio is still limited in the diversity of hand motions, facial expressions, and appearances. Training a model only on such a dataset limits its generalization capability when applied to in-the-wild data. To achieve robust and generalizable hand-face interaction and deformation recovery, we introduce a weak-supervision training pipeline that utilizes in-the-wild images without the reliance on 3D annotations.

In addition to the 2D keypoint supervision for in-the-wild images, we propose a novel depth supervision pipeline.
This pipeline leverages the robust depth prior from a diffusion-based monocular depth estimation model [], which provides essential geometric information for accurate mesh recovery and captures spatial relationships critical for contact state and deformation estimation. To improve our model’s robustness, we further employ pose priors of the hand and face by introducing hand and face parameter discriminators that learn rich hand and face motion priors from multiple datasets on hand or face separately [, ].
By incorporating a small set of real-world images alongside the Decaf dataset and leveraging our weak-supervision pipeline, we markedly enhance the accuracy and generalization capacity of our model.

As a result, our method achieves superior performance in terms of accuracy, physical plausibility, inference speed, and generalizability. It surpasses all previous methods in accuracy on both standard benchmarks and challenging in-the-wild images. Fig. 1 visualizes some results of our method. We conduct extensive experiments to validate our method. In summary, our contribution is three-fold:

- •
We propose DICE, the first end-to-end learning-based approach that accurately recovers hand-face interactions and deformations from a single image.
- •
We propose a novel weak-supervised training scheme with depth supervision on keypoints to augment the Decaf data distribution with a diverse real-world data distribution, significantly improving the generalization ability.
- •
DICE achieves superior reconstruction quality compared to baseline methods while running at an interactive rate (20 fps).

## 2 Related Work

Extensive efforts have been made to recover meshes from monocular images, including human bodies [, , , , , , , , , , , , , , , , , , ], hands [, , , , , , , , , ], and faces [, , , , , , , , , , , ]. This also includes recovering the surrounding environments [, , , , , , , , , ] and interacting objects [, , , , , , , , , , , , , , ] while reconstructing the mesh. The acquired versatile behaviors play a crucial role in various applications, including motion generation [, , , , , , , , , , , , ], augmented reality (AR), virtual reality (VR), and human behavior analysis [, , , , , ]. In the following, we mainly review the related works on hand, face and full-body mesh recovery.

3D Interacting Hands Recovery.
Recent advancements have markedly enhanced the capture and recovery of 3D hand interactions. Early studies have achieved reconstruction of 3D hand-hand interactions utilizing a fitting framework, employing resources such as RGBD sequences [], hand segmentation maps [], and dense matching maps []. The introduction of large-scale datasets for interacting hands [, ] has motivated the development of regression-based approaches. Notably, these include regressing 3D interacting hand directly from monocular RGB images [, , , , ]. Additionally, research has extended to recovering interactions between hands and various objects in the environment, including rigid [, , , , , , , ], articulated [], and deformable [] objects. Following [], our work distinguishes itself by introducing hand interactions with a deformable face, characterized by its non-uniform stiffness—a significant difference from conventional deformable models. This innovation presents unique challenges in accurately modeling interactions.

3D Human Face Recovery.
Research in human face recovery encompasses both optimization-based [, ] and regression-based [, ] methodologies. Beyond mere geometry reconstruction, recent approaches have evolved to incorporate training networks with the integration of differentiable renderers [, , , , ]. These methods estimate variables such as lighting, albedo, and normals to generate facial images and compare them with the monocular input. However, a significant limitation in much of the existing literature is the neglect of the face’s deformable nature and hand-face interactions. Decaf [] represents a pivotal development in this area, attempting to model the complex mimicry of musculature and the underlying skull anatomy through optimization techniques. In contrast, our work introduces a regression-based, end-to-end method for efficient problem-solving, setting a new benchmark in the field.

3D Full-Body Recovery.
The task of monocular human pose and shape estimation involves reconstructing a 3D human body from a single-color image. Optimization-based approaches [, , , ] employ the SMPL model [], fitting it to 2D keypoints detected within the image. Conversely, regression-based methods [, , , , , , , , ] leverage deep neural networks to directly infer the pose and shape parameters of the SMPL model. Hybrid methods [] integrate both optimization and regression techniques, enhancing 3D model supervision. Distinct from these approaches, we follow parametric methods [, , , ] due to its flexibility for animation purposes.
Unlike most research in this domain, which primarily concentrates on the main body with only rough estimations of hands and face, our methodology uniquely accounts for detailed interactions between these components.

## 3 Method

Figure: Figure 2: Overview of the proposed DICE framework. The input image is first fed to a CNN to extract a feature map, which is then passed to the Transformer-based encoders for mesh and interaction, i.e., MeshNet and InteractionNet. The MeshNet extracts hand and face mesh features, which is then used by the Inverse Kinematics models (IKNets) to predict pose and shape parameters that drive FLAME [] and MANO [] models. The InteractionNet predicts per-vertex hand-face contact probabilities and face deform fields from the feature map, the latter is applied to the face mesh output by the FLAME model. To improve the generalization capability, we introduce a Weakly-Supervised Training Scheme using off-the-shelf 2D keypoint detection models [, ] and depth estimation models [] to provide depth supervision on keypoints. In addition, we use head and hand discriminators to constrain the distribution of parameters regressed by IKNets.

Problem Formulation.
Following Decaf [], we adopt the FLAME [] and MANO [] parametric models for hand and face. Given a single RGB image $\mathbf{I}\in\mathbb{R}^{224\times 224\times 3}$, the objective of this task is to reconstruct the vertices of hand mesh $\mathbf{V}_{H}\in\mathbb{R}^{778\times 3}$ and face mesh $\mathbf{V}_{F}\in\mathbb{R}^{5023\times 3}$, along with capturing the face deformation vectors $\mathbf{D}\in\mathbb{R}^{5023\times 3}$ caused by hand-face interaction and its non-rigid nature, and per-vertex contact probabilities of hand $\mathbf{C}_{H}\in\mathbb{R}^{778}$ and face ${\mathbf{C}_{F}}\in\mathbb{R}^{5023}$.

### 3.1 Transformer-based Hand-Face Interaction Recovery

Our model incorporates a two-branch Transformer architecture and integrates inverse-kinematic models, specifically MeshNet, InteractionNet, and IKNets. A differentiable renderer [] is used to compute depth maps from the predicted mesh for depth supervision, and the hand and face discriminators are used as priors for constraining the hand and face poses; See Fig. 2 for an overview.

Given a monocular RGB image $\mathbf{I}$, we use a pretrained HRNet-W64 [] backbone to extract a feature map $\bm{X}_{\text{I}}\in\mathbb{R}^{H\times W\times C}$, where $H,W$ are the spatial dimension and $C$ is the channel dimension. Following [, ], we flatten the image feature maps and upsample the $H\times W$ feature maps to $N$ feature maps, one for each keypoint and coarse vertex of both hand and face. We then concatenate the $\mathbf{F}^{\prime}\in\mathbb{R}^{N\times C}$-dimensional feature maps with a $N\times 3$-dimensional downsampled hand and face vertices and keypoints of the mean pose, as mesh vertices and joints queries for the transformer, resulting in a feature map $\mathbf{F}\in\mathbb{R}^{N\times(C+3)}$. The mesh vertices and joints also serve as the positional encoding. To effectively model the interaction between the vertices, we mask the image feature maps corresponding to a random subset of vertices.

We propose to use two separate branches, MeshNet and InteractionNet, splitting the regression of mesh vertices and a deformation field for their semantic difference: the mesh positions are more global while the deformation vectors and contact states are relatively local. Specifically, the network is followed by two progressively downsampling transformers: MeshNet, which takes the feature map $\mathbf{F}$ as input and regresses the rough vertex positions of hand $\mathbf{V}_{H}^{\prime}$ and face, $\mathbf{V}_{F}^{\prime}$; and InteractionNet, which first downsamples the feature map $\mathbf{F}$, then uses it to predict the 3D deformation field $\mathbf{D}$ at each face vertex along with the contact labels for each hand and face vertices, $\mathbf{C}_{H}$ and ${\mathbf{C}_{F}}$. Note the contacts and deformations are regressed in the same encoder to model their close relationship: the contacts cause the deformations. We validate our design in Sec. 4.4.

Next, instead of directly regressing the hand and face vertices, we regress the pose and shape of the parametric hand and face models, making the output readily animatable for downstream applications. This is achieved by a neural inverse kinematics model, named IKNet, similar to Kolotouros et al. []. Our IKNet takes roughly estimated hand and face mesh vertices $\mathbf{V}_{H}^{\prime}$ and $\mathbf{V}_{F}^{\prime}$ as inputs and predict hand and face pose, shape and expression parameters $(\theta_{\text{h}},\beta_{\text{h}})$, $(\theta_{\text{f-pose}},\beta_{\text{f}},\theta_{\text{f-exp}})$, along with the position and orientation for hand [] and face [], respectively.
We use the predicted parameters to first obtain the hand mesh and undeformed face mesh $\mathbf{V}_{H}$, $\mathbf{V}_{F}^{*}$. Then, we apply the deformation $\mathbf{D}$ predicted by the InteractionNet on $\mathbf{V}_{F}^{*}$ to get the final deformed face $\mathbf{V}_{F}$. Regressing parameters offers several advantages: first, it enables readily animatable meshes; second, compared to non-parametric regression methods, where meshes typically contain artifacts such as spikes [, , ], the mesh quality is significantly improved; third, the compact parameter space facilitates a more effective discriminator, which will be discussed in the following section.

### 3.2 Weakly-Supervised Training Scheme

Although the aforementioned benchmark, Decaf [], accurately captures hand, face, self-contact, and deformations, it consists of eight subjects and is recorded in a controlled environment with green screens. Training a model only with Decaf limits its generalization capability to in-the-wild images that have far more complex and diverse human identities, hand poses, and face poses.

To further enhance the generalization capability of our model, we train our model with $500$ diverse in-the-wild images of hand-face interaction collected from the internet without the reliance on the 3D ground truth annotations. First, we use 2D hand-and-face keypoints detected by [] and [].

Then, we propose to use Marigold [], a diffusion-based monocular depth estimator pre-trained on a large number of images to generate 2D affine-invariant depth maps for supervision in the direction of depth (see Eq. 4). The depth supervision provides a strong depth prior, which guides the spatial relationship between hand and face meshes, promoting accurate modeling of hand-face interaction. For supervision, we use a differentiable rasterizer [] to compute a depth map from the predicted hand and face meshes and supervise the network using a depth loss calculated between the depth values of hand and face keypoints and corresponding points on the predicted depth map. The introduced weak-supervision pipeline significantly enhances our model’s generalization capability and robustness, which we investigate in Sec. 4.4. In our experiment, we found that when training the model with only a small dataset of 500 images, we could significantly improve the model’s accuracy and generalization capability. Moreover, we train adversarial priors on the hand and face parameter space on multiple hand and face pose datasets: the face-only RenderMe-360 [], the hand-only FreiHand [], and Decaf []. This ensures the plausibility of generated face and hand poses and shapes while allowing for flexible poses and shapes beyond the Decaf data distribution to handle in-the-wild cases.

### 3.3 Loss Functions

Mesh losses $\mathcal{L_{\text{mesh}}}$: For richly annotated data in Decaf [], we employ $L_{1}$ loss for 3D keypoints, 3D vertices, and 2D reprojected keypoints against their respective ground-truths, following common practice in human- and hand-mesh recovery [, , ]. We further apply a $L_{1}$ loss $\mathcal{L}_{\text{params}}$ on the estimated hand and face pose, shape, and facial expression against the ground-truth parameters. For in-the-wild data, only the 2D reprojected keypoints are supervised, as this is the only type with corresponding ground truth.

Interaction losses $\mathcal{L_{\text{interaction}}}$: For data in Decaf [], we impose Chamfer Distance losses to enforce touch for predicted contact vertices and discourage collision. We also have a binary cross-entropy loss to supervise contact labels and a deform loss with adaptive weighting to supervise deform vectors, similar to []. For in-the-wild data, we also impose touch and collision losses since they do not require annotations.

Adversarial loss $\mathcal{L}_{\text{adv}}$ are applied to the predicted hand and face parameters for in-the-wild data to constrain their parameter space, and for decaf data to facilitate the training of the discriminators. The adversarial loss is given by:

$$ $\mathcal{L}_{\text{adv}}(E)=\mathbb{E}_{\theta_{F}\sim p_{E}}\left[\log\left(1 -D_{F}(E(I))\right)\right]+\mathbb{E}_{\theta_{H}\sim p_{E}}\left[\log\left(1- D_{H}(E(I))\right)\right].$ (1) $$

The losses for the hand and face discriminators are given by:

$$ $\mathcal{L}_{\text{adv}}(D_{F})=-\left(\mathbb{E}_{\theta_{F}\sim p_{E}}\left[ \log\left(1-D_{F}(E(I))\right)\right]+\mathbb{E}_{\theta_{F}\sim p_{\text{data }}}\left[\log\left(D_{F}(\theta)\right)\right]\right)$ (2) $$

and

$$ $\mathcal{L}_{\text{adv}}(D_{H})=-\left(\mathbb{E}_{\theta_{H}\sim p_{E}}\left[ \log\left(1-D_{H}(E(I))\right)\right]+\mathbb{E}_{\theta_{H}\sim p_{\text{data }}}\left[\log\left(D_{H}(\theta)\right)\right]\right),$ (3) $$

where $E$ jointly denotes the image backbone, the mesh encoder and the parameter regressor, $\theta_{F}=\text{concat}(\theta_{\text{face-shape}},\theta_{\text{face-jaw}},
\theta_{\text{face-exp}})$, $\theta_{H}=\text{concat}(\theta_{\text{hand-shape}},\theta_{\text{hand-pose}})$.

Depth loss $\mathcal{L}_{\text{depth}}$: To provide pseudo-3D hand and face keypoints supervision for in-the-wild data, we use a modified SILog Loss [], an affine-invariant depth loss as our depth supervision $\mathcal{L}_{\text{depth}}$. Formally, let $\hat{K}_{D}$ denote the pseudo-ground-truth affine-invariant depth of the face and hand keypoints, and $K_{D}$ denote the rendered depth for the keypoints,

$$ $\mathcal{L}_{\text{depth}}=\left[\mathbf{Var}\left(\log(K_{D}+\varepsilon)- \log(\hat{K}_{D}+\varepsilon)\right)\right]^{1/2},$ (4) $$

where $\mathbf{Var}$ is the standard variance operator and $\varepsilon=10^{-7}$.

Overall, our loss for the mesh and interaction networks is formulated by

$$ $\mathcal{L}=\lambda_{\text{mesh}}\mathcal{L_{\text{mesh}}}+\lambda_{\text{ interaction}}\mathcal{L_{\text{interaction}}}+\lambda_{\text{adv}}\mathcal{L_{ \text{adv}}}+\lambda_{\text{depth}}\mathcal{L_{\text{depth}}},$ (5) $$

where $\lambda_{\text{mesh}}=12.5,\lambda_{\text{interaction}}=5,\lambda_{\text{depth
}}=2.5,\lambda_{\text{adv}}=1$ for all the experiments in the paper; See more details in Appendix C.

## 4 Experimental Results

### 4.1 Datasets and Metrics

##### Datasets

We employ Decaf [] for reconstructing 3D face and hand interactions with deformations, along with the in-the-wild dataset we collected with 500 images. We use the hand and face shape, pose, and expression data from Decaf [], RenderMe-360 [], and FreiHand [] for training the adversarial priors. We use the training set of the aforementioned datasets for network training. We utilize the Decaf test set for quantitative evaluation, and additionally, we visualize in-the-wild images from the test set for qualitative evaluation.

##### Metrics.

We adopt commonly-used metrics for mesh recovery accuracy following [, , , ]:
$\bullet$ Mean Per-Joint Position Error (MPJPE): the average Euclidean distance between predicted keypoints and ground-truth keypoints.
$\bullet$ PAMPJPE: MPJPE after Procrustes Analysis (PA) alignment.
$\bullet$ Per Vertex Error: per vertex error (PVE) with translation.

Following Decaf [], we use the plausibility metrics mentioned below:
$\bullet$ Collision Distance (Col. Dist.): the average collision distances over vertices and frames;
$\bullet$ Non-Collision Ratio (Non. Col.): the proportion of frames without hand-face collisions;
$\bullet$ Touchness Ratio: the ratio of hand-face contacts among ground truth contacting frames;
$\bullet$ F-Score: the harmonic mean of Non-Collision Ratio and Touchness Ratio.

### 4.2 Implementation Details

We train the MeshNet, InteractionNet, and IKNet along with the face and hand discriminators with three identical AdamW optimizers with a learning rate of $6\times 10^{-4}$, and a learning rate decay of $1\times 10^{-4}$, optimizing in an alternating manner. Our batch size is set to
$16$ during the training stage. Each training task takes $40$ epochs using $48$ hours. The model is trained and evaluated on 8 Nvidia A6000 GPUs with an AMD 128-core CPU. For a fair comparison, all baseline models used the settings in their original papers. Inference times are calculated on a single Nvidia A6000 GPU.

### 4.3 Performance on Hand-Face Interaction and Deformation Recovery

In addition to baselines considered in Decaf [], we compare our method with a representative work in human body/hand mesh recovery, METRO, an end-to-end transformer-based model. For a fair comparison, we compare our method with a modified version of METRO [] for predicting hand and face meshes, with extra output heads added to predict contact and deformation.

Figure: Figure 3: Qualitative results of hand-face interaction, deformation, and contact recovery by DICE on Decaf and in-the-wild images. In contact visualizations, a deeper color indicates a higher contact probability.

Figure: Figure 4: Qualitative comparsion of DICE, Decaf [], PIXIE [], METRO*[] on Decaf validation set and in-the-wild images. Our method achieves superior reconstruction accuracy and plausibility in the Decaf [] dataset, while generalizing well to difficult in-the-wild actions unseen in Decaf.

#### 4.3.1 Quantitative Evaluations

Table: Table 1: Comparison of hand-face interaction and deformation recovery on Decaf.

Reconstruction Accuracy In Tab. 1, our method surpasses all baseline methods in terms of reconstruction accuracy, achieving a $7.5\%$ reduction in per-vertex error compared to the current state-of-the-art, Decaf. Note that our method is regression-based and allows inference at an interactive rate, while Decaf [] uses a cumbersome test-time optimization process, taking more than $700$x more time per image. Decaf also requires using temporal information in successive frames, while our method only uses a single frame. Our method shows a $30\%$ reduction in reconstruction error compared to the modified METRO baseline, and up to $79\%$ reduction compared to other end-to-end baselines.

Plausibility
In addition to high accuracy, our method achieves the highest overall physical plausibility (F-Score) among all regression-based methods. Note that Touchness and Non-Collision ratio are complement to each other and are meaningless when considered individually, while F-Score measures the two values as a whole. Our method has a much lower interpenetration distance (Col. Dist.) compared to Benchmark and PIXIE (hand+face), which consider hand and face separately, therefore generating implausible interactions. Note that PIXIE (whole body) and METRO* show lower collision distances with a much lower Touchness than our method, indicating that the reconstructed hands and faces often appear incorrectly as if they are not interacting. On the other hand, our method shows low collision distance with a high Touchness, indicating plausible hand-face interaction reconstruction.

Contact Estimation In Tab. 2, DICE achieves superior contact estimation performance on Decaf dataset, surpassing previous work [] in F-Score for both face and hand contacts. Here, the F-score provides a comprehensive measure of both the precision and the recall ratio combined. These two metrics are complementary and less meaningful when only considered individually; See Fig. 3 for qualitative results.

**Table 2: Comparison of hand-face interaction and deformation recovery on Decaf.**
| Method | F-score $\uparrow$ | Precision $\uparrow$ | Recall $\uparrow$ | Accuracy$\uparrow$ |
| --- | --- | --- | --- | --- |
| Decaf (face) [] | $0.57$ | $0.69$ | $0.49$ | $0.99$ |
| Decaf (hand) [] | $0.47$ | $0.62$ | $0.39$ | $0.98$ |
| DICE (face) | 0.61 | $0.64$ | $0.57$ | $1.00$ |
| DICE (hand) | 0.50 | $0.55$ | $0.45$ | $0.98$ |

#### 4.3.2 Qualitative Evaluations

As discussed in Sec. 3.2, the Decaf [] dataset is collected in an indoor environment with a green screen, which doesn’t reflect the complex environment where real-world hand-face interactions occur. Therefore, a model only trained with the Decaf dataset might have generalization issues when tested on in-the-wild data.
Fig. 4 confirms this result by our model’s superior generalization performance on in-the-wild data with unseen identity and pose. As shown in Fig. 3, our method faithfully reconstructs hand-face interaction and deformation and accurately labels the area of contact.

### 4.4 Ablation Study

In-the-wild data
As shown in Tab. 3, adding weak-supervision training and in-the-wild data for DICE training improves all reconstruction error metrics (PVE*, MPJPE, PAMPJPE) while maintaining a high plausibility (F-Score).
This is because the limited pose and identity distribution of the Decaf training dataset may cause the model to overfit, and the inclusion of in-the-wild images out of the Decaf data distribution effectively improves the generalization capability of DICE.

Depth Supervision
Although depth supervision is only applied to in-the-wild data, as shown in Tab. 3, removing it also significantly affects performance on the Decaf validation set. Without depth loss, wrong predictions in depth are not penalized for in-the-wild data, introducing noise in the training process, and resulting in erroneous depth predictions in the Decaf dataset.
As shown in Appendix Fig. 7, the absence of depth supervision introduces ambiguity in the z-direction, resulting in artifacts such as self-collision.

Adversarial Prior The adversarial prior incorporates diverse but realistic pose and shape distribution beyond Decaf [], ensuring the reality of regressed mesh while allowing for generalization. As shown in Tab. 3, introducing adversarial supervision improves the accuracy and physical plausibility.

Parameter Supervision Supervising parameters directly, in addition to the indirect supervision of parameters by the mesh losses, improves both plausibility and accuracy. This is because direct parameter supervision eliminates ambiguity, without which the network may resort to other parameter combinations that produce incorrect meshes similar in terms of Euclidean distance.

Intermediate Supervision Removing the constraint that $\mathbf{V}_{F}^{\prime},\mathbf{V}_{H}^{\prime}$ being rough meshes and treating them only as feature maps results in a substantial drop in accuracy and a slight drop in plausibility (F-Score). Also, note that there is a sharp increase in collision distance, which is attributed to the spatial inaccuracy of the final output mesh. This indicates that using an intermediate mesh feature instead of an ordinary feature map increases the spatial accuracy of the output meshes, which also benefits plausibility.

Network Design
In Tab. 3, adopting the two-branch architecture, which separates deformation and interaction estimation from mesh vertices regression, improves both accuracy and plausibility.

### 4.5 Limitations and Future Works

While our method achieves SotA accuracy on the Decaf [] dataset and generalizes well to unseen scenes and in-the-wild cases, we still have failure cases when the hand-pose interactions are extremely challenging and have severe occlusions (see Appendix D.2). Moreover, while our method effectively recovers hand and face meshes with visually plausible deformations, there remains room for improvement in deformation accuracy and physical plausibility. In the future, physics-based simulation [, , , , , ] can be used as a stronger prior, producing more physically accurate estimations. In this paper, although we found using 500 in-the-wild images significantly improves the model’s generalization ability, scaling up to a larger amount of in-the-wild data, on the order of millions or billions, would further enhance performance, which we will study in future work.

Table: Table 3: Comparison of hand-face interaction and deformation recovery on Decaf. Bold denotes the best result.

## 5 Conclusion

In this work, we present DICE, the first end-to-end approach for reconstructing 3D hand and face interaction with deformation from monocular images. Our approach features a two-branch transformer structure, MeshNet and InteractionNet, to model local deform field and global mesh geometry. An inverse-kinematic model, IKNet, is used to output the animatable parametric hand and face meshes. We also proposed a novel weak-supervision training pipeline, using a small amount of in-the-wild images and supervising with a depth prior and an adversarial loss to provide pose priors. Benefitting from our network design and training scheme, DICE demonstrates state-of-the-art accuracy and plausibility, compared with all previous methods. Meanwhile, our method achieves a fast inference speed (20 fps), allowing for more downstream interactive applications. In addition to strong performance on the standard benchmark, DICE also achieves superior generalization performance on in-the-wild data.

## Appendix A Implementation Details

### A.1 CNN Backbone

The CNN backbone used in our framework is an HRNet-W64 [], initialized with ImageNet-pretrained weights. The weights of the backbone would be updated during training. We extract a ($49\times H$)-dim feature map from this network and upsamples it to a ($N\times H$)-dim feature map, where $N=N_{h_{k}}+N_{f_{k}}+N_{h_{v}}+N_{h_{v}}$, the total number of head and hand keypoints $N_{h_{k}},N_{f_{k}}$ and vertices $N_{h_{v}},N_{f_{v}}$. Then, we concatenate the keypoints and the vertices corresponding to the head and hand mean pose as keypoints and vertex queries, resulting in a ($(N+3)\times H$)-dim feature map. Random masking of keypoints and vertex queries of rate $30\%$ is applied, following [].

### A.2 MeshNet and InteractionNet

Our MeshNet and InteractionNet have similar progressive downsampling transformer encoder structures, see Fig. 5 for an illustration. The MeshNet has three component transformer encoders with decreasing feature dimensions. The InteractionNet starts with a fully connected layer that downsamples the feature dimension, followed by two transformer encoders. Each transformer encoder has a Multi-Head Attention module consisting of 4 layers and 4 attention heads. In addition to head and hand mesh features, MeshNet also regresses head and hand keypoints, which are only for supervision and not used by any downstream components.

Figure: Figure 5: Structural details of the MeshNet and InteractionNet. (a) MeshNet; (b) InteractionNet; (c) Internal structure of a Transformer Encoder block.

### A.3 IKNet

Our IKNets take in rough mesh features $\mathbf{V}_{F}^{\prime},\mathbf{V}_{H}^{\prime}$ and output the pose and shape parameters $(\theta,\beta)$, as well as the global rotation and translation $(R,T)$. They feature a Multi-Layer Perceptron (MLP) structure, each consisting of five MLP Blocks and a final fully connected layer. Each MLP Block contains a fully connected layer, followed by a batch normalization layer [] and a ReLU activation layer. There are two skip-connections, connecting the output of the first block with the input of the third block, and the output of the third block with the input of the final fully connected layer. See Fig. 6 for an illustration. The hand and head IKNets have the same structure, differing only in their input and output dimensions. The hidden dimensions of the two IKNets are 1024.

Figure: Figure 6: Structural details of the IKNet.

### A.4 Training and Testing Details

To be consistent with the training setting of Decaf(^1^11Confirmed by the authors of Decaf) [], in the Decaf dataset, we use all eight camera views and the subjects S2, S4, S5, S7, and S8 in the training data split for training. For testing, we use only the front view (view 108) and the subjects S1, S3, and S6 in the testing data split.
The low, mid, and high-resolution head mesh consists of $559$, $1675$, and $5023$ vertices, respectively. The low and high-resolution hand mesh consists of $195$ and $778$ vertices, respectively. We use the middle-resolution head mesh and the high-resolution hand mesh as the inputs of head and hand IKNets.

## Appendix B More Qualitative Comparisons

We demonstrate qualitatively the effect of the absence of the depth loss in Fig. 7. When trained without depth loss, the network is only supervised with 2D information on in-the-wild data, without any constraints in the z-direction. As a result, artifacts such as self-penetration frequently occur in this case. The introduction of depth loss eliminates this ambiguity, allowing the correct relative positioning of hand and face.

Figure: Figure 7: Qualitative demonstration of the effects of the depth loss. The model generalizes poorly in the z-direction when trained without depth supervision.

## Appendix C Addition details on Losses

Here, we provide the details of the mesh losses and the interaction losses. The details of the adversarial loss and the depth loss are already mentioned in the main paper.

### C.1 Mesh losses

The mesh loss $\mathcal{L}_{\text{mesh}}$ consists of four components.

$$ $\mathcal{L}_{\text{mesh}}=\mathcal{L}_{\text{reproj}}+4\mathcal{L}_{\text{vert }}+2\mathcal{L}_{\text{key}}+2\mathcal{L}_{\text{params}}.$ (6) $$

Vertices Loss $L_{1}$ loss is used for predicted rough 3D face and hand vertices ${\mathbf{V}_{f}^{\prime}}$, ${\mathbf{V}_{h}^{\prime}}$, FLAME-regressed undeformed 3D face vertices ${{\mathbf{V}_{f}^{*}}}$ and MANO-regressed 3D hand vertices ${{\mathbf{V}_{h}}}$ against the ground-truth 3D undeformed face vertices $\mathbf{\hat{V}}_{f}$ and 3D hand vertices $\mathbf{\hat{V}}_{h}$.

$$ $\mathcal{L}_{\text{vert}}=\lambda_{h}(\mu_{\text{nonpara}}\|\mathbf{V}_{h}^{ \prime}-\mathbf{\hat{V}}_{h}\|_{1}+\|\mathbf{V}_{h}-\mathbf{\hat{V}}_{h}\|_{1} )+\lambda_{f}(\mu_{\text{nonpara}}\|\mathbf{V}_{f}^{\prime}-\mathbf{\hat{V}}_{ h}\|_{1}+\|\mathbf{V}_{f}^{*}-\mathbf{\hat{V}}_{f}\|_{1}),$ (7) $$

where $\lambda_{h},\lambda_{f}$ are empirically set to $3$ and $1$ respectively. $\mu_{\text{nonpara}}$ is set to $4$ to emphasize the supervision on the more complex non-parametric mesh features.
Keypoints Loss We use $L_{1}$ loss for predicted rough 3D face and hand keypoints $\mathbf{K}_{f}^{\prime}$, $\mathbf{K}_{h}^{\prime}$, 3D face and hand keypoints extracted from rough mesh ${\mathbf{K}_{f_{\text{mesh}}}},{\mathbf{K}_{h_{\text{mesh}}}}$, FLAME-regressed 3D face keypoints ${\mathbf{K}_{f}}$ and MANO-regressed 3D hand keypoints ${\mathbf{K}_{h}}$ against the ground-truth 3D undeformed face keypoints ${\mathbf{\hat{K}}_{f}}$ and 3D hand keypoints ${\mathbf{\hat{K}}_{f}}$.

$$ $\mathcal{L}_{\text{key}}=\mu_{\text{nonpara}}(\|\mathbf{K}_{h}^{\prime}- \mathbf{\hat{K}}_{h}\|_{1}+\|{\mathbf{K}_{h_{\text{mesh}}}}-\mathbf{\hat{K}}_{ h}\|_{1}+\|\mathbf{K}_{f}^{\prime}-\mathbf{\hat{K}}_{f}\|_{1}+\|{\mathbf{K}_{f _{\text{mesh}}}}-\mathbf{\hat{K}}_{f}\|_{1})$ (8) $$

$$ $+\|{\mathbf{K}_{f}}-{\mathbf{\hat{K}}_{f}}\|_{1}+\|{\mathbf{K}_{h}}-{\mathbf{ \hat{K}}_{h}}\|_{1},$ (9) $$

where $\mu_{\text{nonpara}}$ is empirically set to $4$, to put more weight on the non-parametric mesh with high degrees of freedom.
Reprojection loss $L_{1}$ loss is used for reprojected rough 3D face and hand keypoints $\mathbf{K}_{f}^{\prime}$, $\mathbf{K}_{h}^{\prime}$, 3D face and hand keypoints extracted from rough mesh ${\mathbf{K}_{f_{\text{mesh}}}},{\mathbf{K}_{h_{\text{mesh}}}}$, FLAME-regressed 3D face keypoints ${\mathbf{\hat{K}}_{f}}$ and MANO-regressed 3D hand keypoints ${\mathbf{\hat{K}}_{h}}$ against the ground-truth face and hand 2D keypoints $\mathbf{\hat{K}}_{f_{\text{2D}}},\mathbf{\hat{K}}_{h_{\text{2D}}}$.

$$ $\mathcal{L}_{\text{reproj}}=\lambda_{h}(\|\Pi(\mathbf{K}_{h}^{\prime})-\mathbf {\hat{K}}_{h_{\text{2D}}}\|_{1}+\|\Pi({\mathbf{K}_{h_{\text{mesh}}}})-\mathbf{ \hat{K}}_{h_{\text{2D}}}\|_{1}+\|\Pi({\mathbf{K}_{h}})-\mathbf{\hat{K}}_{h_{ \text{2D}}}\|_{1})$ (10) $$

$$ $+\lambda_{f}(\|\Pi(\mathbf{K}_{f}^{\prime})-\mathbf{\hat{K}}_{f_{\text{2D}}}\| _{1}+\|\Pi({\mathbf{K}_{f_{\text{mesh}}}})-\mathbf{\hat{K}}_{f_{\text{2D}}}\|_ {1}+\|\Pi({\mathbf{K}_{f}})-\mathbf{\hat{K}}_{f_{\text{2D}}}\|_{1}),$ (11) $$

where $\Pi$ is the learned camera projection function. $\lambda_{h},\lambda_{f}$ are set to $4$ and $1$ respectively.

Parameter loss
We apply $L_{1}$ loss on the regressed hand and face pose, shape, and facial expression parameters against their respective ground truths.

$$ $\mathcal{L}_{\text{face-params}}=(\|\beta_{\text{f}}-\hat{\beta}_{\text{f}}\|_ {1}+\|\theta_{\text{f-exp}}-\hat{\theta}_{\text{f-exp}}\|_{1}+\|\theta_{\text{ f-pose}}-\hat{\theta}_{\text{f-pose}}\|_{1})/3$ (12) $$

$$ $\mathcal{L}_{\text{hand-params}}=(\|\beta_{\text{h}}-\hat{\beta}_{\text{h}}\|_ {1}+\|\theta_{\text{h}}-\hat{\theta}_{\text{h}}\|_{1})/2$ (13) $$

$$ $\mathcal{L}_{\text{params}}=\mathcal{L}_{\text{face-params}}+\mathcal{L}_{ \text{hand-params}}$ (14) $$

### C.2 Interaction losses

The interaction loss $\mathcal{L}_{\text{interaction}}$ consists of four components:

$$ $\mathcal{L}_{\text{interaction}}=0.2\mathcal{L}_{\text{touch}}+0.6\mathcal{L}_ {\text{contact}}+\mathcal{L}_{\text{collision}}+6\mathcal{L}_{\text{deform}}.$ (15) $$

Deformation loss
Due to the human anatomy, some vertices on the face are more easily deformed than other vertices. Therefore, we impose an adaptive weighting on each vertex and use square loss to penalize large deformation. We also have a regularization term to penalize extremely large deformations.

$$ $\mathcal{L}_{\text{deform}}=\sum_{i\in\mathcal{I}}(1+\mu\|\hat{d_{i}}\|_{2})\| \hat{d_{i}}-d_{i}\|_{2}^{2}+\lambda\sum_{i\in\mathcal{L}}\|d_{i}\|,$ (16) $$

where $\mathcal{I}$ is the set of indices of face vertices, $d_{i}$, $\hat{d_{i}}$ are the predicted and ground truth deformation vector for index $i$, and $\mathcal{L}=\{i\in\mathcal{I}:\|d_{i}\|_{2}>3cm\}$ the vertices of large deformations. $\mu$ and $\lambda$ are empirically set to be $5000$, $100$ respectively.

Touch loss
Let $\mathbf{V}_{F_{C}}$ and $\mathbf{V}_{H_{C}}$ denote the set of face and hand vertices that are predicted by the model to have contact probability greater than $0.5$.

$$ $\mathcal{L}_{\text{touch}}=\text{CD}(\mathbf{V}_{F_{C}},\mathbf{V}_{H_{C}})+ \text{CD}(\mathbf{V}_{H_{C}},\mathbf{V}_{F_{C}}),$ (17) $$

where $\text{CD}(X,Y)$ gives the mean Chamfer Distance (CD) between each point in $X$ to the closest point in $Y$.

Collision loss
Let $\mathbf{V}_{H_{\text{Col}}}$ denote the set of hand vertices that penetrates the face surface, $\mathbf{V}_{F}$ and $\mathbf{D}_{F}$ denote the predicted face mesh vertices and deformations.

$$ $\mathcal{L}_{\text{collision}}=\text{CD}(\mathbf{V}_{H_{\text{Col}}},\mathbf{V }_{F}-\mathbf{D}_{F}).$ (18) $$

Contact loss
Let $\mathbf{C}_{H}$ and $\mathbf{C}_{F}$ denote the predicted hand and face contact probabilities and ${\mathbf{\hat{C}}_{H}}$, ${\mathbf{\hat{C}}_{F}}$ denote the ground-truth contact labels.

$$ $\mathcal{L}_{\text{contact}}=\text{BCE}(\mathbf{C}_{H},{\mathbf{\hat{C}}_{H}}) +\text{BCE}({\mathbf{C}_{F}},{\mathbf{\hat{C}}_{F}}),$ (19) $$

where BCE denotes the binary cross-entropy loss.

## Appendix D More Discussions

### D.1 Performance under Challenging Occlusion.

Figure: Figure 8: Examples of failed keypoint estimation in case of large self-occlusion. (a) input image; (b) inaccurate keypoint estimation by the same keypoint estimators used in Decaf [, ]; (c) reconstructed hand-face interaction by our method. (d) reconstructed hand-face interaction by Decaf.

As seen in Fig. 8, our end-to-end DICE method is robust under challenging self-occlusion cases, such as the hand covering more than half of the face. On the other hand, Decaf [], which requires an initial keypoint prediction for test-time optimization, performs poorly in this situation.

### D.2 Failure Cases

In Fig. 9, we demonstrate the failure cases of our method. As shown in Fig. 9 (a), when there is a complex interaction between the hand and face, such as the presence of a cleaning sponge, there is a drop in the reconstruction accuracy of the hand mesh recovery. Also, as in Fig. 9 (b), When the face completely occludes the hand, a highly challenging scenario unseen in the training data, our model could not faithfully reconstruct the hand position.

Figure: Figure 9: Samples of failure cases.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Is it possible to apply depth supervision to other reconstruction tasks? What makes it particularly suited to the hand-face problem, aside from data availability? Could this approach be more broadly relevant to visual tasks involving contact points between two objects? I wonder if this could be explored further.
soundness: 3
strengths: -- The use of depth supervision for weakly-supervised training in mesh reconstruction is innovative and potentially impactful for similar tasks.
summary: This paper presents a model for the simultaneous reconstruction of face and hand meshes from a single image, based on previous work and annotated data from Shimada et al. (DECAF). The model employs a Transformer-based image encoder to extract keypoints for face and hand through separate streams, utilizing parametric models for reconstruction. Designed for images that contain both a face and a hand, the approach extends training with unannotated data by leveraging a diffusion-based depth estimation model (Ke et al., 2024) to guide mesh reconstruction. After reconstructing the face and hand mesh, the depth maps are computed and aligned with depth estimates from the diffusion model. The model is trained with four types of losses: mesh loss and contact-point loss, similar to those in DECAF, as well as depth loss and adversarial loss, which support the depth supervision process. Experimental results show improvements over previous work for face-hand interaction images.
weaknesses: -- The model is focused specifically on face-hand images, which may limit its relevance to the broader computer vision community. The emphasis on this specific interaction appears to be largely application-driven, leaving questions about its applicability to other computer vision tasks.

-- Although the use of depth supervision in training mesh models appears novel, it leverages known components, which slightly reduces the overall technical novelty.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Explain the notable gap in Decaf regarding physics plausibility metrics.
soundness: 3
strengths: * The motivation is compelling. DICE provides the first end-to-end learning-based method for capturing hand-face interactions and deformations from a single image, filling a significant gap in the field.
* Good experimental results. The paper claims that DICE achieves superior reconstruction quality compared to baseline methods while operating at an interactive rate (20 fps), which is crucial for real-time applications.
* Technically sound. The weakly-supervised training scheme using in-the-wild images enhances the model's ability to generalize beyond the constraints of studio-collected data.
summary: The paper introduces a novel method for reconstructing 3D hand-face interactions with deformations from a single image. The authors present DICE, a Transformer-based architecture that simultaneously estimates hand and face poses, contacts, and deformations. A key feature of DICE is its ability to disentangle local deformation fields from global mesh vertex locations into two separate network branches, which enhances the accuracy of deformation and contact estimation for precise hand-face mesh recovery. Additionally, the method utilizes a weakly-supervised training approach with in-the-wild images that lack 3D ground-truth annotations, thereby improving the model's generalizability. Experiments show state-of-the-art performance in terms of accuracy and physical plausibility, along with interactive inference rates on modern GPUs.
weaknesses: * DICE is better than optimization-based methods like Decaf. However, there is a notable gap in physics plausibility metrics compared to Decaf. Can the authors provide a clearer explanation? I wonder if it struggles to balance reconstruction accuracy and physics plausibility.
* In Table 3, using in-the-wild data negatively impacts the F-score. The authors should provide a clearer explanation.
* The novelty is limited. The techniques used in this paper are commonly employed in the community.
* The authors are encouraged to include more discussion on failure cases.
* Some results are not convincing. In Figure 4, the head reconstruction of Decaf is noticeably better than that of the authors' method for the third sample from in-the-wild data.
* Some videos should be included to demonstrate real-time capability.

---

confidence: 5
contribution: 2
initial_rating: 6
presentation: 3
questions: _see **Weaknesses** for key questions/remarks._


### Q1. Hand Rigidity $\rightarrow$ Applicability of CONTHO
- One limitation of this method, as well as prior Decaf, is to only consider face deformation and ignore hand one. While the hands are definitely _stiffer_, ignoring their soft tissues can yield incorrect results in some cases (e.g., hand pressed against forehead). Moreover, by considering the hand as piece-wise rigid, the target task becomes similar to the much less under-explored domain of human/object interaction modeling (i.e., both tasks can be expressed as modeling 3D non-rigid/rigid interactions). It seems that recent solutions such as CONTHO [Nam et al., 2024] could be adapted to face/hand reconstruction. What is the authors' position w.r.t. these adjacent methods? How would the authors differentiate their own approach?
soundness: 3
strengths: _(somewhat ordered from most to least important)_

### S1. Clear Explanations and Decent Reproducibility
- The Methodology is clear, thoroughly explained, and overall well-formalized. 
- While the actual implementation has not been provided, an expert in the art should be able to re-implement this work.
- The paper is also well illustrated and easy to follow (pipeline figures + qualitative results).
- Limitations are well-discussed, with failure cases provided (Appendix D).

### S3. Convincing Evaluation
- The evaluation is also clear, comprehensive, and well presented (e.g., highlighting optimization-based vs. regression-based methods in Table 1, providing various qualitative results, etc.).
- The proposed DICE method outperforms the SOTA on 3D reconstruction metrics, at a competitive running time.
- Qualitative results and the ablation study are also convincing. In-the-wild examples help illustrate the generalizability of the method.

### S3. End-to-End Solution Targeting an Under-explored Task
- As mentioned above, this paper is seemingly only the second (after Decaf [Shimada et al., 2023]) to address the joint modeling of hand/face geometry and interaction in a comprehensive, end-to-end manner. As such, it would be valuable to the community.
- While the proposed method mostly relies on existing components (CNN-backbone, Transformer processing, weak supervision using large monocular depth regressor, etc.), it holds value as a well-designed end-to-end system.
summary: ### Motivation
- Modeling the geometry and interactions of human hands and face from a monocular image is an underexplored tasks in 3DCV, with broad applications (AR/VR, motion capture, etc.).
- To the best of the authors' knowledge (and mine), only one prior method tackle this problem in an exhaustive and end-to-end manner, Decaf [Shimada et al., 2023]. This seminal work is however optimization-based, and therefore too slow for interactive use.

### Contributions
- The authors thus introduce DICE, which performs the above-mentioned task in a forward-inference manner. The method relies on a Transformer-based network, with two branches separately tackling the regression of (a) the hand+face geometry and (b) the face deformation caused by hand contact.
- Furthermore, the authors introduce a weak supervision to expand training to in-the-wild-ish images (2D keypoint annotations are still needed, though not 3D ones). Depth supervision is also enabled on such data by integrating a pre-trained monocular depth estimator to generate pseudo ground-truth. Predicted meshes are projected into depth images using a differentiable renderer for comparison.


### Results
- Experiments on the Decaf benchmark show that DICE outperforms prior art in terms of 3D reconstruction accuracy and provides competitive runtime on par with regression-based baselines (~0.1s, whereas Decaf needs ~15s). Unlike prior art though, the authors did not fully disclose their results on physics-plausibility relating to touchness and collision.
- An ablation study, as well as plenty of convincing qualitative results, are also provided. 

### Relevance
- This paper addresses a relevant yet under-explored 3DCV task. The community would benefit from this iteration over the seminal Decaf.
weaknesses: _(somewhat ordered from most to least important)_


### W1. Limited Technical Novelty
- The proposed solution has value as an end-to-end system tackling an under-explored task, but its technical novelty is limited.
- The main/only technical contribution claimed by the authors are their weak depth supervision module, relying on a large/foundation model [L110-112]. There might be some novelty in applying such a model to this specific task (e.g., in terms of processing the depth-map predictions to supervise the facial keypoints), but the underlying idea of leveraging a large monocular depth model for weak supervision is not novel.
- Most of the loss functions appear borrowed from Decaf [Shimada et al., 2023], even though this is not made obvious in the paper (both in Section 3.3. and in Appendix C).



### W2. Lack of Insight w.r.t. Physics Metrics
- Unlike Decaf, the authors here did not report all physics-related metrics (non-collision / touchness accuracy).
- The authors could provide more insight in terms of the challenges faced by regression-based methods for accurate modeling, compared to optimization-based solutions.
- Similarly, Table 2 and corresponding paragraph [L417-421] lack proper insight/description. The superiority of DICE is not as obvious as claimed.

### W3. Somewhat Limited Comparison to Prior Art
- The authors compare their method to 3 other solutions: Decaf [Shimada et al., 2023] (most relevant as it also targets hand/face interaction), PIXIE [Feng et al., 2021a] and METRO [Lin et al., 2021a)] (both originally developed for full-body regression), and a baseline/benchmark solution [Lugaresi et al., 2019; Li et al., 2017)] (composed of two independent hand-only and face-only models). While the literature does indeed lack in prior art focusing on hand/face interaction, more baseline methods could have been considered, e.g., mixing other single-target methods (MinimalHand , RingNet, FastMETRO, etc.). This could have further highlighted the benefits on modeling both body parts jointly.
- To counterbalance this "weakness", it should be noted that the aforementioned set of prior methods have been entirely borrowed from Decaf, so the authors here stuck to Decaf's protocol, which is fair. More comparisons could make the paper stronger.

### W4. Minor Remarks
- Societal impact is not discussed, showing a lack of self-reflection. While positive applications are actually listed in the Introduction (e.g., benefits to AR/VR and gaming industry), a variety of negative applications can also come to mind: surveillance (appearance + gesture), animatable deepfakes, etc. Discussions on data fairness/bias are also always relevant when it comes to human-modeling solutions.

Meta-review: This work introduces an end-to-end method for capturing hand-face interaction deformations from a single image. Accurately recovering hand-face interactions with plausible deformations is significant due to its broad applications in AR/VR.
The proposed approach includes three key innovations:
	1.	Utilizing an attention mechanism to effectively model the relationship between the hand and face.
	2.	Dividing the regression process into two branches to disentangle global geometry from local interaction.
	3.	Learning an intermediate non-parametric mesh representation instead of directly regressing hand and face parameters.
As a novel task, the method is compared with the state-of-the-art Decaf, demonstrating superior performance and improved inference efficiency. 

Three reviewers have provided positive feedback on this work, though concerns were raised about its technical novelty, as it integrates existing techniques. The AC has reviewed the paper and discussion during rebuttal period, acknowledging this work's efforts in addressing hand-face deformations from a single image. The pipeline, composed of multiple modules, is well-documented, with insights into each design clarified by the authors. The AC encourages the release of the codes and datasets used in this work to further contribute to this emerging field.

Decision: Accept (Poster)