Learning Dense Hand Contact Estimation from Imbalanced Data
Authors: Daniel Sungho Jung
Kyoung Mu Lee
Venue: NeurIPS 2025 poster ()
Keywords: Hand contact estimation
Hand-object interaction
Primary area: applications
TL;DR: We propose HACO, a framework for dense hand contact estimation that addresses class and spatial imbalance challenges in training on large-scale datasets.

Abstract: Hands are essential to human interaction, and exploring contact between hands and the world can promote comprehensive understanding of their function. Recently, there have been growing number of hand interaction datasets that cover interaction with object, other hand, scene, and body. Despite the significance of the task and increasing high-quality data, how to effectively learn dense hand contact estimation remains largely underexplored. There are two major challenges for learning dense hand contact estimation. First, there exists class imbalance issue from hand contact datasets where majority of regions are not in contact. Second, hand contact datasets contain spatial imbalance issue with most of hand contact exhibited in finger tips, resulting in challenges for generalization towards contacts in other hand regions. To tackle these issues, we present a framework that learns dense HAnd COntact estimation (HACO) from imbalanced data. To resolve the class imbalance issue, we introduce balanced contact sampling, which builds and samples from multiple sampling groups that fairly represent diverse contact statistics for both contact and non-contact vertices. Moreover, to address the spatial imbalance issue, we propose vertex-level class-balanced (VCB) loss, which incorporates spatially varying contact distribution by separately reweighting loss contribution of each vertex based on its contact frequency across dataset. As a result, we effectively learn to predict dense hand contact estimation with large-scale hand contact data without suffering from class and spatial imbalance issue. The codes are available at https://github.com/dqj5182/HACO_RELEASE.

Learning Dense Hand Contact Estimation from Imbalanced Data
Daniel Sungho Jung' Kyoung Mu Lee
IPAI, DDept. of ECE & ASRI, Seoul National University, Korea {dqj5182, kyoungmu) Senu ac.kr
Abstract
Hands essential human interaction, and exploring contact between hands and the world can promote comprehensive understanding of their function. Recently, there have been growing number of hand interaction datasets that cover interaction with object, other hand, scene, and body. Despite the significance of the task and increasing high-quality data, how to effectively learn dense hand contact estimation remains largely underexplored. There are two major challenges for learning dense hand contact estimation. First, there exists class imbalance issue from hand contact datasets where majority of regions are not in contact. Second, hand contact datasets contain spatial imbalance issue with most of hand contact exhibited in finger tips, resulting in challenges for generalization towards contacts in other hand regions. To tackle these issues, we present framework that learns dense HAnd COntact estimation (HACO) from imbalanced data. To resolve the class imbalance issue, we introduce balanced contact sampling, which builds and samples from multiple sampling groups that fairly represent diverse contact statistics for both contact and non-contact vertices. Moreover, to address the spatial imbalance issue, we propose vertex-level class-balanced (VCB) loss, which incorporates spatially varying contact distribution by separately reweighting loss contribution cach vertex based on its contact frequency across dataset. As result, we effectively learn to predict dense hand contact estimation with large-scale hand contact data without suffering from class and spatial imbalance issue. The codes available at http:///ggttuuuuoo/mggaa/g/// com/dqj5182/HHAO HACC RELEASE
Introduction
From infancy, humans rely on physical contact to perceive and interact with the surrounding environ- ment. Among the various mediums of contact, hands play a predominant role, facilitating essential tasks and enabling effective communication. Consequently, developing robust hand contact estima- tion model is crucial for advancing our understanding of hand interactions and addressing various challenges
35) that require accurate hand contact estimation.
Over the recent years, we have witnessed significant advances in large-scale interaction datasets involving hands. Hand-object interaction datasets [19,
36 24 focused on capturing 41 considered two hand interaction from single person. Hand-face interaction dataset (55] investigated on interaction between hand and
15
16
hand grasps of objects. Hand-hand interaction datasets [59
22 58 extended the scope of interaction to include environments featuring the ground, walls, and large-scale objects. Hand-body interaction dataset [65] covered broad range of interaction with human body as part of numan-humaa interaction. Despite the wide range of diversity the hand interaction datasets with contact annotations [19 5
deformable face. Hand-scene interaction datasets 18,
15 [6] [2 36) 24 41 [5 (55 17 22 65 62), to the best of our knowledge, there has been no attempt to build hand contact estimation model trained on such diverse hand contact. Based on such diverse sets of hand contact data, we aim to build an effective dense hand contact estimation model.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

There are two major issues for training dense hand contact estimation. First, there exists class
(o) Class imbalance issue of hand contact datasets
imbalance between contact and non-contact class. Human participants of the hand in- teraction datasets are often instructed to per- form specific actions such as picking up an ob- and pointing to specific regions of objects 24 or other hands [11. As most of such actions require precise control with interac- tion by a small region on hand surface, most of the hand regions overlooked and excluded from being in contact. In Figure Ta we provide comparison the occurrence of contact and non-contact in major hand contact datasets for
ject [5
15
36
DexYCa
Intertand2.0H
H0L4D
HOOD
Decat
ARCTIC
Spatial imbalance issue of hand contact datasets
all vertices of MANO hand model [54]. This ob- servation shows severe class imbalance issue of hand contact from existing hand contact datasets, Numerically, YCB [5] dataset has roughly 2.7:1 ratio of non-contact contact vertices. More severe InterHand2. 2.6M [41] dataset has huge imbalance ratio of 19.5:1. Such imbalance is also shown in Decaf [55] dataset with 21.7:1 1:1
DoxYCs
IrlstHandd..0
HOMD
HO3D
Decat
ARCTIC
ratio. As stated by numerous studies on data imbalance 3 6 52 26] such data imbalance causes poor performance on underrepresented classes, which is in fact the contacting cases in our scenario. Second, the spatial imbalance is- sue of hand contact data exists throughout the majority of hand contact datasets. This imbal- ance arises from the fact that hand interaction predominantly involves the fingertips, which provide high degree of freedom in both move- ment and rotation. Due to such high maneuverability and the precise nature of actions from motion capture datasets, most of hand contact from the datasets are significantly skewed finger tip regions. As shown in Figure lb] the heatmap for hand contact from hand contact datasets are highly concen- trated the finger tips. Due to such spatial imbalance issue of hand contact datasets, models directly trained on the datasets struggle to generalize on diverse contact patterns across hand.
Figure 1:
Two challenges for dense hand contact estimation in the wild. First, hand contact datasets suffer from class imhalance, the majority of vertices contain contact. Second, spatial imbalance arises because contact points are predominantly concentrated at the fingertips. Due to these issues, models trained on such data struggle to generalize to diverse contact patterns across the hand.
Therefore, we propose HACO, framework that learns dense hand contact estimation from imbal- anced data. Initially, HACO is trained on a large-scale, assorted dataset composed of 14 datasets, shown in Table I that contains various hand interactions to leverage the power of large-scale train- However, this large-scale dataset still exhibits inherent imbalance in terms of both class and spatial distribution. To address these issues, we introduce two strategies designed to fully exploit the potential of such large-scale data. First, we introduce Balanced Contact Sampling (BCS), which mitigates class imbalance in hand contact datasets by dividing the dataset into multiple sampling groups, each representing different statistics of contact and non-contact. To ensure fair representation of the contact statistics, we compute contact balance score that measures how much each hand contact instance deviates from the dataset-wide average. This metric enables us to assess whether each instance adjust the sampling ratios accordingly, ensuring all contact types are well represented. Second, we (VCB) loss, which applies spatially varying weights each
ing 22 5 23
64
49
mnder-representeer or over-represented within the original dataset and to
present the Vertex-Level vertex to address the spatial imbalance issue in hand contact datasets. Inspired by the core idea of
(CB) loss (6), which re-weights the loss based on class frequency, we adapt this concept to the spatial domain by computing separate loss weight for each vertex based on how well contact and non-contact are represented at the vertex within the dataset. While the original CB loss applies global class-level weighting, our method performs fine-grained, vertex-level weighting, allowing the model to focus on represtee contact patterns a spatially manner.

---

Table 1: Dataset configuration. We leverage 14 datasets with various hand interaction.
Dataset
Interaction
Domain
# of images
# subjects
# objects
ObMan 19
Rigid Object
Synthetic
152K
20
8
DexYCB
Rigid Object
Real Indoor
582K
10
20
MOW
Rigid Object
Real Outdoor
0.5K
450
121
HO3D 15
Rigid Object
Rcal Indoor
78K
10
H203D [6
Rigid Object Hand
Real Indoor
76K
10
ARCTIC 12
Articulated Object Hand
Real Indoor / Studio
2. IM
10
11
HOI4D 36
Rigid Object + Hand
Real Indoor
2.4M
16
Rigid Object + Hand
Real Indoor
571K
InterHand2.66
Hand
Real Indoor Studio
2.6M
HIC 59
Hand
Real / Indoor
3GK
PROX 17
Secne
Real Indoor Studio
100K
20
RICH
Scene
Real / Outdoor
540K
Decaf
Face (Wholc-Body) Whole-Body
Real Indoor
100K
Hi4D
Real Indoor Studio
11K
As a result, HACO achieves state-of-the-art performance across diverse hand contact scenarios, in- cluding hand-object, hand-hand, hand-scene, and hand-body interactions, consistently outperforming existing methods.
Our key contributions are follows:
We introduce HACO, novel framework that addresses the data imbalance problems in hand contact datasets and enables effective learning dense hand contact estimation from large-scale data.
To mitigate class imbalance in existing hand contact datasets, we propose Balanced Contact Sampling (BCS), which constructs multiple sampling groups fairly represent diverse contact statistics.
To handle spatial imhalance, we present the Vertex-Level Class- Balanced (VCB) loss, which computes vertex-specific weighting factor based the dataset-wide contact distribution.
In the end, HACO, the first method trained for dense hand contact estimation large-scale dataset, achieves substantial performance gains across diverse contact scenarios.
2
Related works
Dense hand contact estimation. Most of the existing methods that explore dense hand contact
estimation are devoted dense human-scene contact estimation [18] 22 58 43). Given an image of
human-scent interaction, such methods estimate contactness for each vertex of 3D human model (c.g.. SMPL (37), SMPL-X [48). POSA [18] based upon the idea that the human-scenc contact is largely influenced by the 3D pose of human. Hence, the work employs a conditional variational
autoencoder framework 56 to learn human-scene contact with the 3D vertex position of posed human as condition. BSTRO 2 takes another approach for estimating human-scees contact, which directly relies on visual input with Transformer- based architecture [9] To train the model, they proposed RICH dataset [22] that annotates dense human-scene contact by capturing 3D geometry of body and scene with multi-view laser scanner in both indoor and outdoor environment. Lastly, to improve generalization ability of human-scene contact estimation, DECO
(58 focused how
annotate human-scene contact the wild. They introduced simple yet effective contact annotation method that simply asks crowd-soureed annotators to paint 3D human model (i.c.. SMPL (37) in neutral T-pose, which significantly enhanced the potential of scaling contact data. Despite numerous attempts dense human-scene contact estimation, there lacks in-depth investigation contact estimation. Furthermore, we found that dense hand contact estimation has unique challenges compared to human-scena contact estimation. First, due class imbalance of hand contact datasets, the distribution of the datasets are often skewed towards non-contact class, making it non-trivial to train hand contact estimation from datasets. Second, there exists spatial imbalance issue of hand contact datasets where majority of hand contact occurs in finger tips with insufficient data for hand contact in rest of the regions (e-g., dorsum). We introduce HACO, dense hand contact estimation method that effectively tackles such data imbalance issues of hand contact datasets.
dense hand

---

Sett Fill-tttmtion
Tranffomet
Selt Aettrtnn Transfomar
Emtralding
Ores Attundon
Crisk Atterdon
Trandfomar
Transforen
ViT
Patmar
Liveet
Eppood
sakan
Embed
image
Oedea
Output
Transformer
Figure 2: Overall pipeline of HACO. Our method encodes input image as image tokens with a ViT backbone after patch embedding layers. Given the image tokens along with positional embeddings and a contact token, multiple layers of self-attention Transformer and cross- Pross-atenntion Transformer produce an output token. Lastly, the output token is passed through linear layer and combined with the contact initialization, followed by a sigmoid activation to output the final hand contact prediction.
3D hand and object interaction reasoning. Dense hand contact reveals key spatial and semantic relationship between 3D hand and object. Initially, dense hand contact was studied in the field of grasping with specific tasks such as grasp contact prediction [T] contact-guided hand and object pose refinement [14]. and contact-conditione grasp generation (34] Such works made tremendous impact on robotics, especially on how robots can naturally grasp an object. However, their dense hand contact mostly predicted from 3D geometry of hand and object without exploring how directly predict the dense hand contact from visual input. Recently, the field of 3D hand and object reconstruction showed potential the effectiveness of contact for improving both reconstruction accuracy and natural interaction between hand and object. ContactOpt [14] proposed a differentiable contact optimization that optimizes hand and object poses inferred from image-based methods with estimated contact to achieve natural interaction. EasyHOI (35] introduced prior-guided hand-object interaction optimizer that refines initial 3D hand and object using heuristically determined contact regions to improve interaction quality. Our HACO is applicable to these downstream tasks as long as visual input is available. To demonstrate HACO's effectiivness, we show that our predicted dense contact improves 3D grasp optimization over geometry drivenn contact estimation from ContactOpt [14), and 3D reconstruction over the previously used heuristic contact regions in EasyHOI 355
Method
3.1 Model architecture
Given an RGB image I E p3xHxW where H and W denote the height and width of the image, we first embed the image into multiple tokens with a patch embedding layer and process these tokens using a Vision Transformer (ViT) [1Q. Following ViTPose (63] and HaMeR [49], we reshape the tokens and formulate the image feature F E R 1280x 16x Additionally, we build a contact token, which acts as query token for hand contact estimation. To inject image feature F into the contact token, we process the image feature and positional embedding with consecutive self-attention Transformer and cross-attention Transformer with contact token query and image feature F as key- value pair. Afterwards, the output contact token is processed with linear layer. order to resemble residual layers (20 that stabilize training, we add contact initialization to the output of linear layer. learnable embedding that naturally learns the most effective initial contact estimation during training. Lastly, sigmoid layer converts the final contact logits C e R into the estimated dense hand contact probabilities, where V 778 MANO hand vertices [54]
The contact initialization
3.2 Balanced contact sampling
A hand contact dataset D {H, (vi, c:)l consists of N hand instances, where each H; contains the 3D hand vertices v, E IR Vx3 and binary vertex contact vector c; E 10, As illustrated Figure Tal the dataset D exhibits a strong class imbalance, with many hand instances lacking contact. To alleviate this issue, we propose balanced contact contlln (BCS), which constructs sampling bins B- = [Bi, .B...., Bx consisting of K groups of hand instances. Each bin B, is formed by binning samples based their contact balance scores s,, which reflect how cach hand's

---

contact C deviates from the dataset- wide average e. To compute s;, we first define the dataset-wide contact mean as с & Ei= C, which represents the contact probability each vertex across all samples. The contact balance score s; for each sample is then defined as:
S (c c) c)
where higher scores correspond to hand instances H; whose contact patterns deviate more from the dataset-wide average, enabling balanced grouping across diverse contact statisties that span both contact and non-contact samples. Based on the distribution of contact balance scores {s; (5)/=1 we partition the dataset into K bins B,..... B5 by applying non-linear binning. We first compute the minimum and maximum contact balance scores as 8min min; 8; and 8max = max; 8;, and define K+1 bin edges using logarithmic spacing scheme controlled by a curvature parameter 3 >> 0 (e.g..
B = 5):
log(1 + 3-3 xK) log(1 +
Smin +
($max
Smin)
K k K.
(2)
This results in bin edges (TO, Ti...., Tk} that allocate finer resolution higher contact scores. Each group B& is then constructed by selecting hand instances whose contact balance scores fall within the corresponding interval:
B₁ = HHe E D s E (Tk-1.
Due to the skewed distribution of scores 8;, the bins B₂ may differ in size. To equalize their contribution, we apply stratified resampling to ensure each bin contains the same number of hand instances in the final training set. In the end, this binning strategy enables more accurate grouping and fair sampling of contact-rich hand instances that are underrepresented in the original dataset D, improving the representational balance of the sampling bins B.
3.3 Vertex-level class-balance loss
As illustrated in Figure lb hand contact datasets exhibit severe spatial imbalance, with contact heavily concentrated at the fingertips. To address this issue, we propose a vertex-level class-balanced (VCB) loss, which builds upon the formulate hand contact estimation as a binary classification problem. A simple binary cross-entropy (BCE) loss is defined b p) y log(p) (1 y) log(1 p), where y E (0, 1} is the ground-trute truth label and [0, 1] is the predicted probability. When applied over all hand vertices V, the overall BCE loss becomes:
(CB) sigmoid binary cross- entsopy loss [6]. We first
LBCE =
Σ VEV
M
where we denote ground-truth contact for vertex for vertex u as p.. = (2)) with sigmoid function as a and predicted logit as 2.
yo E 10.1) and predicted contact probability
From the BCE loss, class-balanced (CB) loss 6 introduces a weighting factor for each class assigning greater importance to under-represented classes. In our case, the class e corresponds the binary contact label y E {0, 1}. Accordingly, we denote the class-balanced weighting factor as ogy Simply, we can write CB loss as follows:
1
LCB
Σ
(5)
Specifically, the weighting factor a, is defined using the effective number of samples E(!) where
1
1-3 I 1-9
0g
(6)
(2)
Here, ny denotes the number occurrences class y. and B E [0, 1) is hyperparameter that controls the influence of sample count on the weighting. When the contact class y appears frequently in the dataset, the contact frequency value ny becomes large, which increases the effective number E(v) T-33 and results in smaller class-balanced weight a, E: As result, the CB loss down- weights the contribution of frequent classes and helps mitigate class imbalance in dense hand contact datasets.
1-8"y
5

---

Nevertheless, CB loss 66] is defined at the class level, based on the binary contact label E {0, 1}. If we directly apply CB loss, the weighting factor ay takes only two distinct values, one for contact and one for non-contact, and is applied uniformly across all spatial regions. In the Dex YCB dataset [S]. contact and non-contact vertices account for approximately 76.86% and 23.14% of all vertex labels, respectively. Accordingly, the weighting factor ay is computed using sample counts ny = (0.7686 x n, 0.2314 x rt), where n is the total number contact labels in the dataset. However, this count By is applied uniformly across all vertices, without considering spatial variation. For example, fingertip vertices are much more likely to be in contact, while dorsal hand vertices are
rarely in contact. As a result, the class- baass-balanced weight dy and effective sample number E(v) do not accurately reflect the local distribution of contact and non-contact each vertex. To address this limitation, extend the CB loss formulation to operate at the vertex level.
We introduce the vertex-level class-balanced (VCB) loss, which computes spatially varying weight- ing factor Oy and effective number of samples E( for each vertex v. To account for the spatial imbalance in hand contact datasets, simply assign a separate weighting factor Gy each vertex, depending on the local distribution of contact labels. Given the number of samples Ny. for contact class y at vertex v, the weighting factor is computed as:
y
E(Ea)
Bray
Using this spatially varying weight, the vertex-level blassbbalanced loss is defined as:
CvCB =
Σ NEV
Oyy Po).
(8)
V
This spatial adaptivity enables the VCB loss effectively mitigate the spatial imbalance inherent in hand contact datasets. The VCB loss is applied through progressive weighting strategy that combines the CB loss and VCB loss throughout training. At the beginning of training, only the CB loss is applied, while the weight of the VCB component increases linearly with each epoch. The VCB weight reaches a fixed maximum at the final epoch, resulting a smooth transition from global to vertex- aartev-adapiive supervision. Lastly, we include a lightweight false-positive regularization term to suppress overly confident contact predictions for non-contact vertices.
3.4 Final outputs and loss functions
The final dense hand contact output supports multi-level supervision through projections to coarser resolutions. From the dense contact logits C regressed by the final linear layer, we build multi-level hand contact predictions C' = {C, TH' x CI for N = 4 and E {778, 336, 84, 21} where J.t E R is a regressor that maps full MANO vertices into coarse representations. We then apply a sigmoid layer to convert the logits into contact probabilities. To train HACO, we apply three loss functions: the vertex-level class balacce (VCB) loss for multi-level hand contact supervision, regularization loss, and smoothness loss. Please refer Section 33.3 for details on the VCB loss, and to the Appendix for the regularization and smoothness losses. Briefly, the regularization loss is defined as the L1 loss between the dense contact prediction and the wide mean ground-truth contact, encouraging the prediction to stay close to the average. The smoothness loss measures how predicted contact and non-contact regions are spatially disconnected. This encourages HACO to predict a small number of large contact regions rather than many small, fragmented ones. We set the weights of the VCB loss, regularization loss, and smoothness loss to 1, 0.1, and 1, respectively.
4 Implementation details
PyTorch 47 is used for implementation. Our backbone is initialized with the pre-trained weights of publicly released HaMeR [49]. Following HaMeR, we apply data augmentations including random scaling, cropping, and rotation. To improve robustness to degraded inputs, we additionally perform low-resolution, noise, and blur augmentations. We use the Adam optimizer (38) with a learning rate of 10-5 and mini-batch size of 24. For stable convergence, the learning rate is reduced by a factor of 0.9 after 5 and 10 epochs. We train HACO for 10 epochs on single NVIDIA A6000 GPU.

---

Experiments
5.1 Datasets
We select 14 datasets with diverse hand interactions including ObMan 19]. Dex [.. MOW [4]
HO3D [15]. H203D [16]. ARCTIC [12], HOI4D [36]. H2O (24 for hand-object interaction, In-
terHand2.6M 41, HIC 59 for hand-hand interaction, PROX [77. RICH [22] for hand-scene
interaction, and Decaf (55), Hi4D (65) for hand-body interaction. To reduce redundancy from large video datasets, we employ sampling ratio of 5, 10, 5 for HOI4D [36.. InterHand2.6M [41], and Decaf [55] dataset, respectively. Note that this is separate from the balanced contact sampling in Section 3.2 For ARCTIC [12] dataset, we select data captured from an egocentric viewpoint, while for HOI4D 36]. we use data involving rigid objects due their uniqueness. Since the 3D hand annotations in PROX [17] are inaccurate in the qualitative split, we use only the quantitative subset. For the RICH dataset, we follow the official split used BSTRO [22] for fair comparison. In total, this results in 655K images with ground-truth dense hand contact labels. For details on our method on extracting ground-truth dense hand contact labels, please refer Appendix.
5.2 Evaluation metries
To evaluate dense hand contact estimation, we compute precision, recall, and Fi-score. However, recall and Fi-score are undefined for fully non-contact samples. Thus, skip fully non-contacting hands during evaluation. We also assess contact quality on two downstream tasks: 3D hand grasp optimization [14] and 3D hand-object reconstruction 351 For 3D hand grasp optimization [14, we evaluate Intersection Volume (Inter Vol.), Mean Per-Joint Position Error (MPJPE), and contact metrics. For 3D hand-object reconstruction, we compare HACO using Mean Per- Vertex Position Error (MPVPE), MPJPE, CDbo. F-5ho- and F-10po against the contact method of EasyHOL. F-5ho and F-10 denote F-scores of reconstructed 3D hand and object using 5mm and 10mm thresholds.
5.3 Ablation study
Effectiveness of balanced contact sampling, In Table 2 the proposed balanced contact sampling strategy significantly enhances the performance across all metrics. Compared to the model trained without sampling, our method improves 1.0% in precision, 12.0% in recall, and 8.5% Fl Fl-score. The substantial boost in recall, in particular, highlights the effectiveness of balanced sam- pling in mitigating class imbalance by exposing the model to more positive contact instances. Overall, these results demonstrate that pling strategy plays a critical role in improving our model's ' ability to detect contact accurately and comprehensively.
Table 2: Ablation on balanced contact sampling strategy on MOW 4 dataset.
Methods
Precision? Recallf Fl Fl-Scoref
w/o sampling
0.520
0.542 0.607
0.481 0.522
w/ sampling (Ours)
0.525
Effectiveness of vevtex-level class-balanced loss. In Table 3 the proposed VCB loss achieves the best overall performance by effectively addressing data imbalance at a finer granularity. Unlike CB loss, which applies class bal- ancing at the hand-level by weighting pos- itive and negative samples across all ver- tices, VCB loss operates at the per-vertex level, allowing the model to learn contact patterns from individual vertex more accu- rately. This vertex-wise weighting enables the model to better capture subtle and spa- tially varying contact signals, especially under-represented regions, leading to im- proved recall without sacrificing precision. As a result, VCB loss yields the highest F1-score among all loss-based methods, demonstrating that resolving class imbal- ance locally, rather than globally, is key to effective dense contact estimation.
Table 3:
Comparison of various loss-based tech- niques for data imbalance on MOW 4 dataset.
Methods
Precision
Recallf
F1 F1-Scoref
CE loss L11 loss
0.530
0.294
0.348
0.521
0.392
0.413
1.2 loss
0.531
0.298
0.352
Focal loss
0.518
0.387
0.409
CB loss 6
0.484
0.534
0.465
CB Focal loss
6
0,522
0.360
0.392
LDAM loss g
0.532
0.224
0.293
Asymmetric loss
0,484
0.479
0.440
Poly loss 26
0,528
0.324
0,371
VCB loss (Ours)
0,525
0.607
0,522
7

---

Table 4: Comparison of various training dataset configurations on MOW 4 HIC [59],
RICH [22], Hi4D [65]. HO: Hand-object, HH: Hand-hand, HS: Hand-scene, HB: Hand-body.
Test dataset
Train dataset
Precision f
Recall f FI-Score
HS HS+HH
0.397
0.516
0.411
0,405
0,420
0.373
MOW
HS+HH+HB
0.425
0.551
0,408
HS+HH-+HB+HO (Ours)
0.525
0.607
0.522
HS
0.188
0.231
0,187
HS+HO
0.094
0.260
0.134
HIC 52
HS+HO+HB
0.114
0.265
0.158
HS+HO+HB++H (Ours)
0.216
0.409
0.263
HO
0.551
0.547
0.492
НO+HI
0.554
0,522
0.471
RICH 22
НО
0.554
0.581
0.490
НОННIIНIBBI (Ours)
0.741
0.899
0.781
HS
0,515
0,439
0.419
HS+HO
0.610
0.503
0.496
Hi4D 65
HS+HO+HH
0.596
0.513
0.498
HS+HO+НIН사НB (Ours)
0.555
0.636
0.565
Effectiveness of large-scale data configuration. Our final model benefits significantly from the inclusion of all four interaction types: hand-object (HO), hand-hand (HH), hand-body body (HB), and hand-scene (HS). As shown in Table 4 performance improves consistently across all evaluated datasets as more interaction types are incrementally incorporated, with the full combination achieving the highest overall F1-score. Removing HO leads to substantial drop in recall, further highlighting its essential role in capturing diverse contact patterns. Although HB and HH alone contribute modestly, their addition enhances Fl-scores and stabilizes overall performance, indicating their complementary effects. Omitting HS results in reduced Fl-score despite strong contributions from the other components, suggesting importance in separating hand contacts from background context. Ultimately, only the full configuration achieves strong and balanced performance across all metrics and datasets, demonstrating that each interaction type contributes uniquely and that leveraging the full diversity is crucial for learning accurate and generalizable contact representations.
5.4 Comparison with state-of-the-ee methods
Dense hand contact estimation. Table presents a comparison between our HACO and state- of-the-art approaches of POSA [18]. BSTRO [22). and DECO (58) on the MOW 4 dataset.
Our method achieves substantial improvements
Table 5: Comparison with SOTA methods of hand contact estimation on MOW 4 dataset.
across all metrics, significantly outperforming the strongest prior method, DECO (58). POSA (18] shows limited performance due its sole reliance on pose priors 13] without image evidence, while BSTRO (22 suffers from the lack of large-scale, di- verse contact data and does not address class or spa- tial imbalance during training. DECO
Methods
Precisiont
Recallf F1 F1-Scoref
POSA 18
0.134
0.128
0.101
58 improved over earlier methods, but performance remains con- strained by the lack of diverse supervision and mech- anisms for handling imbalance. In contrast, HACO addresses these challenges by training on diverse contact configurations, initializing contact pre- diction with learned prior, and tackling class and spatial imbalance through targeted loss design. Figure3furtth further highlights HACO's s qualitative superiority, showing more precise and anatomically plausible contact patterns across scenarios, such as fingertip and palmar contact when holding mic, partial palmar contact when dicing with knife, and thumb-index finger contact during grasp of pencil. Overall, our method shows superior performance over prior SOTA methods, demonstrating its potential as strong model for dense hand contact estimation by addressing key challenges that limit earlier approaches.
BSTRO 22
0.204
0.126
0.112
DECO 58
0.246
0.235
0.197
IACCO (Ours)
0.525
0.607
0.522

---

Imput
Dunsal
Polker
Dunnl
Palnar
Danal
Pakner
Durol
POSA
BSTRO
DECO
HACO (Ours)
Ground-truth
Figure 3: Qualitative comparison of dense hand contact estimation with POSA 18), BSTRO [22),
DECO [58] on MOW [ dataset. We highlight exemplar regions where HACO outperforms previous methods. Note that we only predict right hand contact.
3D hand grasp optimization. Table 6 compares our HACO with the prior state-of- the-art method, DeepContact [14,, for contact-guided 3D grasp optimization using predicted hand-object poses from HFL-Net 33 on the DexYCH [5] dataset. Despite DeepContact having access to full 3D hand and object meshes for contact estimation, method achieves comparable superior results using only image input. HACO attains lower MPJPE, which indicates more accurate hand articulation. Further- more, achieves higher precision, recall, and F1-score, demonstrating improved contact quality and coverage. Overall, these results show that HACO enables more effective and robust contact-aware grasp optimization, without access full 3D hand and object geometry, highlighting its practical applicability real-world scenarios.
Table 6: Comparison of various contact estimation methods for 3D hand grasp optimization DexYCB [5] dataset.
Methods
Inter Vol.
MPJPEJ Precisiont
Recallt FI-Scoref
DeepContact
[4]
29.175
37.155
0.522
0.830
0.612
HACO (Ours)
29.264
3D hand and object reconstruction. Table compares the performance EasyHO1 35 between using its original contact estimation module and using HACO, keeping all other components changed. This isolates the impact of the contact on reconstruction performance. HACO outperforms the original across all metries, including MPVPE, MPJPE, Chamfer Distance (CDno), and F-scores. These consistent gains suggest that HACO produces more accurate and physically plausible contact predictions, directly improving hand-object alignment and reconstruction quality. Figure further supports this, showing that the 3D) reconstruction with contact from HACO more precisely captures interaction between the hand and the pen, unlike EasyHOl's heuristic-based approach.
Table 7: Comparison of various contact estima- tion methods for 3D hand and object reconstruc- tion with HOI (35) on MOW [4] dataset. PVE and PJE refer to MPVPE and MPJPE on vertices and joints between estimated and ground- truth 3D hand, respectively.
Input image
EasyHOI
HACO (Ours)
PVEL
PJEL
CDyp
F.5
F-10,
EasyHOl t5
21.254
20.973
8.338
0.120
0.230
Figure 4: Qualitative comparison of 3D hand- object reconstruction on MOW [4] dataset.
TIACO (Ouns)
21.093
20.845
8.186
0.122
0.231

---

Conclusion
We propose HACO, novel and powerful method that learns dense hand contact estimation from imbalanced data. For class imbalance and spatial imbalance in hand contact datasets, we propose balanced contact sampling and vertex-level previous methods by significant margin on dense hand contact estimation and shows effectiveness on grasp optimization and hand and object reconstruction.
bass-bbllaneed loss. As result, our HACO outperforms
Acknowledgements
This work supported in part by the IITP grants [No.2021-0-01343] No.2023-0-00156, and Artificial Intelligence Graduate School Program (Seoul National University) 1-222-0-00068), and the Industrial Technology Alchemist Project wwo.Rs-2024-00/4 funded by MOTIE, Korea.
10

---

References
[I] Samarth Brahmbhatt, Cusuh Ham, Charles Kemp, and James Hays. ContactDB: Analyzing and predicting grasp contact via thermal imaging. In CVPR, 2019.
[2] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Vosse Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In NeurlPS, 2020.
[3] Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with abbel-distribution-aware margin loss. In NeurlPS, 2019.
[4] Zhe Cao, Iija Radosavovic, Angjoo Kanazawa, and Jitendra Malik. Reconstructing hand-object interactions in the wild. In ICCV, 2021.
[5] Yu-Wei Chao, Wei Yang, Yu Xiang, Pavlo Molchanov, Ankur Handa, Jonathan Tremblay, Yashraj S Narang, Karl Van Wyk, Umar lqbal, Stan Birchfield, Jan Kautz, and Dieter Fox. Dex YC: A benchmark for capturing hand grasping of objects. In CVPR, 2021.
[6] Yin Cui, Menglin Jia, Tsung- Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number of samples. In CVPR, 2019.
[7] Dawgor-Haggerty et al. Trimesh.
[8] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In CVPR, 2009.
[9] Jacob Devlin, Ming-Wey Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional Transformers for language understanding. In NAACL, 2019.
[10] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
[11] Zicong Fan, Adrian Spurr. Muhammed Kocabas, Siyu Tang, Michael Black, and Otmar Hilliges. Learning to disambiguate strongly interacting hands via probabilistic per-pixel part segmentation. In 3DV, 2021.
[12] Zicong Fan, Omid Taheri, Dimitrios Tzionas, Muhammed Kocabas, Manuel Kaufmann, Michael J Black, and Otmar Hilliges. ARCTIC: A dataset for dexterous bimanual hand-object manipulation. In CVPR, 2023.
[13] Yao Feng, Vasileios Choutas, Timo Bolkart, Dimitrios Tzionas, and Michael J Black. Collabo- rative regression of expressive bodies using moderation. In 3DV, 2021.
[14] Patrick Grady, Chengcheng Tang, Christopher D) Twigg, Minh Vo, Samarth Brahmbhatt, and Charles C Kemp. ContactOpt: Optimizing contact to improve grasps. In CVPR, 2021.
[15] Shreyas Hampali, Mahdi Rad, Markus Oberweger, and Vincent Lepetit. HOnnotate: A method for 3D annotation of hand and object poses. In CVPR, 2020.
[16] Shreyas Hampali, Sayan Deb Sarkar, Mahdi Rad, and Vincent Lepetit. Keypoint Transformer: Solving joint identification challenging hands and object interactions for accurate 3D pose estimation. In CVPR, 2022.
[17] Mohamed Hassan, Vasileios Choutas, Dimitrios Tzionas, and Michael Black. Resolving 3D human pose ambiguities with 3D scene constraints. In ICCV, 2019.
[18] Mohamed Hassan, Partha Ghosh, Joachim Tesch, Dimitrios Tzionas, and Michael Black. Populating 3D scenes by learning human-scene interaction. In CVPR, 2021.

---

[19] Yana Hasson, Gul Varol, Dimitrios Tzionas, Igor Kalevatykh, Michael J Black, Ivan Laptev, and Cordelia Schmid. Learning joint reconstruction of hands and manipulated objects. CVPR, 2019.
[20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[21] Junxing Hu, Hongwen Zhang, Zerui Chen, Mengcheng Li, Yunlong Wang, Yebin Liu, and Zhenan Sun. Learning explicit contact for implicit reconstruction of hand-held objects from monocular images. In AAAI, 2024.
[22] Chun-Hao P Huang, Hongwei Yi, Markus Höschle, Matvey Safroshkin, Tsvetelina Alexiadis, Senya Polikovsky, Daniel Scharstein, and Michael Black. Capturing and inferring dense full-body human-scene contact. In CVPR, 2022.
[23] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment Anything. In ICCV, 2023.
[24] Taein Kwon, Bugra Tekin, Jan Stühmer, Federica Bogo, and Mare Pollefeys. H2O: Two hands manipulating objects for first person interaction recognition. In ICCV, 2021.
[25] Jaerin Lee, Daniel Sungho Jung, Kanggeon Lee, and Kyoung Mu Lee, SemanticDraw: Towards real-time interactive content creation from image diffusion models. In CVPR, 2025.
[26] Zhaoqi Leng, Mingxing Tan, Chenxi Liu, Ekin Dogus Cubuk, Jay Shi, Shuyang Cheng, and Dragomir Anguelov. PolyLoss: A polynomial expansion perspective of classification loss functions. In ICLR, 2022.
[27] Jiefeng Li, Siyuan Bian, Chao Xu, Zhicun Chen, Lixin Yang, and Cewu Lu. HybrlK-X. Hybrid analytical-neural inverse kinematics for whole-body mesh recovery. TPAMI, 2025.
[28] Zhihao Li, Jianzhuang Liu, Zhensong Zhang, Songeen Xu, and Youliang Yan. CLIFF: Carrying location information in full frames into human pose and shape estimation. In ECCV, 2022.
[29] Kevin Lin, Lijuan Wang, and Zicheng Liu. fnd-to-end human pose and mesh reconstruction with Transformers. In CVPR, 2021.
[30] Kevin Lin, Lijuan Wang, and Zicheng Liu. Mesh graphormer. In /CCV, 2021.
[31] Tsung-Yo Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In CVPR, 2017.
[32] Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In ICCV, 2017.
[33] Zhifeng Lin, Changxing Ding, Huan Yao, Zengsheng Kuang, and Shaoli Huang. Harmonious feature learning for interactive hand-object pose estimation. In CVPR, 2023.
[34] Shaowei Liu, Yang Zhou, Jimei Yang. Saurabh Gupta, and Shenlong Wang. ContactGen: Generative contact modeling for grasp generation. ICCV, 2023.
[35] Yumeng Liu, Xiaoxiao Long, Zemin Yang, Yuan Liu, Mare Habermann, Christian Theobalt, Yuexin Ma, and Wenping Wang. EasyHOl: Unleashing the power of large models for recon- structing hand-object interactions in the wild. In CVPR, 2025.
[36] Yunze Liu, Yun Liu, Che Jiang, Kangbo Lyu, Weikang Wan, Hao Shen, Boqiang Liang, Zhoujic Fu, He Wang, and Li Yi. HOI4D: A 4D egocentric dataset for category-level human-object interaction. In CVPR, 2022.
[37] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. SMPL: A skinned multi-person linear model. ACM TOG, 2015.
[38] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
12

---

[39] Junzhe Lu, Jing Lin, Hongkun Dou, Ailing Zeng, Yue Deng, Xian Liu, Zhongang Cai, Lei Yang. Yulun Zhang. Haoqian Wang, and Ziwei Liu. DPoser-X: Diffusion model as robust 3D whole-body human pose prior. In ICCV, 2025.
[40] Gyeongsik Moon, Shunsuke Saito, Weipeng Xu, Rohan Joshi, Julia Buffalini, Harley Bellan, Nicholas Rosen, Jesse Richardson, Mallorie Mize, Philippe de Bree, Tomas Simon, Bo Peng. Shubham Garg, Kevyn McPhail, and Takaaki Shiratori. A dataset of relighted 3D interacting hands. In NeurlPS, 2023.
[41] Gyeongsik Moon, Shoou-I Yu, He Wen, Takaaki Shiratori, and Kyoung Mu Lee. InterHand2.6M.: dataset and baseline for 3D interacting hand pose estimation from a single RGB image. In ECCV, 2020.
[42] Lea Muller, Ahmed AA Osman, Siyu Tang, Chun-Hao P Huang, and Michael Black. On self-contact and human pose. CVPR, 2021.
[43] Hyeongjin Nam, Daniel Sungho Jung, Gyeongsik Moon, and Kyoung Mu Lee. Joint reconstruc- tion of 3D human and object via contact-based refinement Transformer. In CVPR, 2024.
[44] Hyeongjin Nam, Daniel Sungho Jung, Yeonguk Oh, and Kyoung Mu Lee. Cyclic test-time adaptation on monocular video for 3D human mesh reconstruction. In ICCV, 2023.
[45] JoonKyu Park, Daniel Sungho Jung, Gyeongsik Moon, and Kyoung Mu Lee. Extract-and- adaptation network for 3D interacting hand mesh recovery. In ICCV, 2023.
[46] JoonKyu Park, Yeonguk Oh, Gyeongsik Moon, Hongsuk Choi, and Kyoung Mu Lee. HandOcc- Net: Occlusion-robust 3D hand mesh estimation network. In CVPR, 2022.
[47] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang. Zach Vetito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, high-performance deep learning library. In NeurIPS, 2019.
[48] Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed AA Osman, Dimitrios Tzionas, and Michael Black. Expressive body capture: 3D hands, face, and body from a single image. In CVPR, 2019.
[49] Georgios Pavlakos, Dandan Shan, Ilija Radosavovic, Angjoo Kanazawa, David Fouhey, and Jitendra Malik. Reconstructing hands in 3D with Transformers. In CVPR, 2024.
[50] Aditya Prakash, Benjamin Lundell, Dmitry Andreychuk. David Forsyth. Saurabh Gupta, and Harpreet Sawhney. How do i do that? Synthesizing 3D hand motion and contacts for everyday interactions. In CVPR, 2025.
[51] Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Yuan Wu, Ross Girshick. Piotr Dollár, and Christoph Feichtenhofer. SAM 2: Segment Anything in images and videos. In ICLR, 2025.
[52] Tal Ridnik, Emanuel Ben-Baruch, Nadav Zamir, Asaf Noy, Itamar Friedman, Matan Protter, and Lihi Zelnik-Manor. Asymmetric loss for multi-label classification. In ICCV, 2021.
[53] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
[54] Javier Romero, Dimitrios Tzionas, and Michael J Black. Embodied hands: Modeling and capturing hands and bodies together. ACM TOG, 2017.
[55] Soshi Shimada, Vladislav Golyanik, Patrick Pérez, and Christian Theobalt. Decaf: Monocular deformation capture for face and hand interactions. ACM TOG, 2023.
[56] Kibyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. In NeurIPS, 2015.
13

---

[57] Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang. Deep high-resolution representation learning for human pose estimation. In CVPR, 2019.
[58] Shashank Tripathi, Agniv Chatterjee, Jean-Claude Passy, Hongwei Yi, Dimitrios Tzionas, and Michael Black. DECO: Dense estimation of 3D human-scene contact in the wild. In ICCV, 2023.
[59] Dimitrios Tzionas, Luca Ballan, Abhilash Srikantha, Pablo Aponte, Mare Pollefeys, and Juergen Gall. Capturing hands in action using discriminative salient points and physics simulation. IJCV, 2016.
[60] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017.
[61] Qingxuan Wu, Zhiyang Dou, Sirui Xu, Soshi Shimada, Chen Wang, Zhengming Yu, Yuan Liu, Cheng Lin, Zeyu Cao, Taku Komura, Vladislav Golyanik, Christian Theobalt, Wenping Wang, and Lingjie Liu. DICE: End-to-end deformation capture of hand-face interactions from single image. In ICLR, 2025.
[62] Xianghui Xie, Xi Wang, Nikos Athanasiou, Bharat Lal Bhatnagar, Chun-Hao P. Huang, Kaichun Mo, Hao Chen, Xia Jia, Zerui Zhang. Liangxian Cui, Xiao Lin, Bingqiao Qian, Jie Xiao, Wenfei Yang, Hyeongjin Nam, Daniel Sungho Jung, Kihoon Kim, Kyoung Mu Lee, Otmar Hilliges, and Gerard Pons-Moll. RHOBIN Challenge: Reconstruction of human object interaction. arXiv preprint 21di::2401.0144444 2024.
[63] Yufei Xu, Jing Zhang, Qiming Zhang, and Dacheng Tao. ViTPose: Simple Vision Transformer baselines for human pose estimation. In NeurlPS, 2022.
[64] Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth Anything: Unleashing the power large-scale unlabeled data. In CVPR, 2024.
[65] Yifei Yin, Chen Guo, Manuel Kaufmann, Juan Jose Zarate, Jie Song, and Oumar Hilliges. Hi4D:
4D instance segmentation of close human interaction. In CVPR, 2023.
[66] Hongwen Zhang, Yating Tian, Yuxiang Zhang, Mengcheng Li, Liang An, Zhenan Sun, and Yebin Liu. PyMAF-X: Towards well-aligned full-body model regression from monocular images. TPAMI, 2023.
[67] Qitao Zhao, Ce Zheng, Mengyuan Liu, Pichao Wang, and Chen Chen. PoseFormerV2: Explor- ing frequency domain for efficient and robust 3D human pose estimation. In CVPR, 2023.
14

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificatin right after your answer (even for NA).
The checklist answers
integral part of your paper submission. They visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]" ", is perfectly acceptable to answer "[No] provvided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). ). general, answering "[No] " or "[NA]" is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: We clearly state the core motivation understand dense hand contact, outlines the challenges of class and spatial imbalance, introduces the proposed methods including balanced contact sampling and vertex-level flass-balanced loss, and presents the contribution of a framework for learning dense hand contact estimation from imbalanced data. These claims are addressed in the introduction and supported by experimental results, accurately reflecting the paper's 's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals not attained by the paper.
15

---

2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The limitations are discussed in the appendix, where we address the scope and boundaries of our method, including future directions.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specinf well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: Our paper does not contain any theoretical result.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
16

---

Answer: [Yes]
Justification: The paper provides all necessary details to reproduce the main experiments, including data preprocessing, model architecture, training settings, and evaluation protocols.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, it may be that access to the model is limited in some way (e.g.. registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: In the abstract of the paper, we provide URL our publicly available code in an open repository.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cc.
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https! for more details.
//nips //ipps.cc/publll http:cccppppccc/tettette.e...clt http://ullggguude/eeeeee /at.uddeee/ddduuuuuasssooooolooo
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
17

---

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
The paper specifies dataset splits, model architecture, training schedules, optimizer settings, and other key hyperparameters necessary understand and reproduce the results.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: The results are obtained from standard randomly seeded runs without cherry- picking, offering a fair and representative evaluation despite not including error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
18

---

Answer: [Yes]
Justification: The paper specifies the type of GPU used, batch size, number of training epochs, and key runtime settings needed to reproduce the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e-g.. preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The research adheres to the NeurlPS Code of Ethics, with no use of sensitive data or methods posing foreseeable ethical risks.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification:
The paper discusses societal impacts in the appendix, including both potential benefits and risks.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out, For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
19

---

11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The work does not involve releasing models or data with high risk of misuse, specific safeguards are not applicable.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All external datasets and code- bases used in the paper are properly cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[NA]
The paper does not introduce new assets such as datasets that require separate documentation.
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
20

---

The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
We mainly utilize publicly available datasets for our experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: We mainly utilize publicly available datasets for our experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMSs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: LLM is not used in any critical component of the research process for this paper.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer LLM policy for what should or should not be described.
21

---

A
Appendix
In this appendix, we provide additional technical details and experimental results that were omitted from the main manuscript due to space constraints, The contents are summarized below:
Details of loss functions
Details of contact initialization
Details of dense hand contact labels
A.4
More examples of class imbalance issue
A.5
More examples of spatial imbalance issue
A.6
More comparisons with state-of-the-ar methods
More ablation studies
A.8
Quantitative results with different backbones
A.9
Computational requirements
A.10
More qualitative results
A.11
Limitations and societal impacts
A.1 Details of loss functions
Smoothness loss. We penalize spatially isolated fragmented contact predictions by leveraging the mesh topology of the hand. Let Pa E [0, 1] denote the predicted contact probability after the sigmoid for each vertex u E V, and let A (0, VVV be sparse adjacency matrix representing
vertex connectivity in the mesh. Here, Aru = indicates that vertex is connected to vertex and ( otherwise. For numerical stability, we also include ceff-connections so that each vertex connected to itself. For each vertex v E V, we compute the aggregated contact prediction over its neighbors as:
Pr
Aon Pus WEV
Σ
and similarly compute the aggregated non-contact prediction as:
Σ WEV
Ave (1 -pw),
(10)
where P. and ÿ represent the neighborhood-smoother estimates for contact and non-contact proba- bilities, respectively. To maintain consistent scale across batches, we further normalize the aggregated contact and non-contact maps by their maximum values within each batch before computing the isolation score. To assess spatial inconsistency, we compute the discrepancy between each vertex's s prediction and its neighborhood average. Specifically, define the isolation score as:
Sy pp - P. [[1 p.) qul-
(11)
This term penalizes contact predictions that sharply differ from those of their neighbors, whether in the contact or non-contact region. To prevent the isolation score from disproportionately penalizing spatially large confident regions, normalize the total discrepancy by the of neighborhood weights:
71p
A
A - Po QV:
(12)
BEV
which corresponds to the degree of vertex in the mesh. This normalization accounts for how many vertices contribute to the isolation score at each location, ensuring that the loss measures average inconsistency rather than accumulating errors over larger neighborhoods. As result, the model is encouraged produce smooth contact maps without being biased against large spatially coherent contact regions. We then define the final smoothness regularization loss as:
Evev Sy Liso log (1+ Eney +e
(13)
22

---

where is small constant added for numerical stability. This loss encourages smooth transitions and penalizes sharp discontinuities in the prediction map. By suppressing highly localized contact signals and promoting coherent spatial clusters, it guides the model to favor fewer but larger contact regions, better reflecting real-world hand interaction patterns.
Regularization loss. To prevent the model from predicting arbitrarily skewed contact distributions, we introduce global regularization loss that encourages predictions to stay close the dataset-wide mean contact pattern. Let Pa E [0, 1] denote the predicted contact probability after the sigmoid for each vertex E V, and let Pa E [0, 1] denote the kataset-wide average contact probability at vertex v, computed over all training samples. We then define the regularization loss as:
Crog
Σ VEV
Pv P
(14)
M
which an LI loss between the predicted contact probabilities and the mean distribution. This encourages the model to avoid degenerate or overly confident predictions that deviate significantly from typical contact patterns, providing a soft global constraint during training.
A.2 Details of contact initialization
To stabilize the training of HACO on dense hand contact estimation, we propose a contact initial- ization technique that utilizes a learnable embedding to learn the most effective initial contact for dense hand contact estimation. In Table AT, our proposed contact initialization outperforms all other initialization methods, achieving the superior performance in precision, recall, and Fl-score. Unlike other approaches that rely on fixed priors or simplistic assumptions. our method introduces a contact representation learned from input data, enabling more accurate and robust initialization during inference. This formulation captures the most useful initial prediction compared to static or handcrafted methods. leading to gains across all metrics. Such consistent improvement demonstrates that learning contact directly from large-scale datasets, rather than relying pre-defined numbers or dataset averages, is crucial for precise contact estimation.
Table A1: Comparison of various contact initialization on MOW 4 dataset.
Methods
Precision
Recallf
FI-Scoref
No initialization
0.508
0.589
0.503
No contact
0.514
0.532
0.482
Full contact
0.511
0,551
0.492
Mean of Dex DCB 5
0.520
0.588
0.512
Mean of MOW 4
0.511
0.600
0.515
Mean of IH26M 41
0.522
0.567
0.505
Ours
0.525
0.607
0.522
A.3 Details of dense hand contact labels
Following the previous works on dense human-scens contact estimation 18, 22,22, 43
we employ
bissance-based thresholding to generate ground-trutt truth dense hand contact labels. This is implemented using the Trimesh library (7). Given ground-truth hand mesh and the mesh of the interacting entity (i.e., an object for hand-object interaction, another hand for hand-hand interaction, a face for hand-face interaction, the environment for hand-scene interaction, or the body for hand-body interaction), we construct a proximity query from the interacting mesh and compute the closest surface point on it for each vertex of the hand mesh. Contact is then determined by thresholding the Euclidean distance between each hand vertex and its corresponding closest point. The resulting contact labels are binary values per vertex, indicating whether each vertex is in contact.
To set these thresholds, we manually inspected the ground-truth meshes and the resulting contact labels produced under fixed threshold values. We observed that the accuracy of 3D annotations varies significantly across datasets, making a single threshold unsuitable for all cases. uniform threshold often led to incorrect labeling and semantically implausible contact regions. Based on our analysis, we set the contact distance threshold to cm for the following datasets: ObMan 19). Dex YCB 55)
HO3D [15), H203D (16), ARCTIC [12]. HO14D [36), H2O 24], and PROX 17 For the MOW 4
23

---

20,020,220
20,076,136
274,398
383,092
D44
Total Cours
3294
Total CRurte
Totee
12M4
7,486,308
2,747 .666
094
Contact
Mon Contact
Contact
Mon-Gontact
DH
Contact
Non ConTACT
0M
Contact
Ron-Contact
H2O
H203D
HIC
MOW
323,60 853
3234
32.141,022 B2Z
92.076, 700
3014
Total Count
60M
19.121,477
13M
14,033
4,371,274
17.107,00 032
Contact
Mon-Contact
Contact
Non Contact
Contact
Contact
Contact
Non-Contact
ОВMan
PROX
RICH
Hi4D
Figure Al: More examples of class imbalance between contact and non-contact H2O (24). H203D [16], HIC (59), MOW [4) ObMan [9), PROX (17), RICH [22), Hi4D 665)
dataset, we used threshold of 3.5 em due to its lower mesh fidelity. For InterHand2.61, and HIC [59], we adopted smaller threshold of 0.5 em to better capture fine-graines hand-hand interactions.
Although most datasets produced reasonable contact labels with a cm threshold, MOW required a larger value of 3.5 due coarse mesh annotations generated through optimization in-the-wild images. For hand-hand interaction datasets, the cm threshold, which is approximately the width of finger, often resulted in inaccurate and semantically implausible contact labels. To address this, adopted a smaller threshold of 0.5 cm. For datasets with existing annotated contact labels, such as directly used the provided ground-truth annotations.
RICH 22
Decaf [51, and Hi4D 655
A.4 More examples of class imbalance issue
Figure AI presents the class imbalance issue across datasets 24 16 59
22 65 omitted
from the main paper due space constraints. In general, all datasets exhibit severe class imbalance. with the majority of vertices corresponding to non-contact. Notably, some datasets such as H2O and HIC display highly skewed ratios of 10.57:1 and 58.41:1, respectively, between non-contact and contact vertices. It is important to note that contact and non-contact counts are computed at the vertex-level rather than per-hand instance. Since the ground-trute contact labels are provided per-hand, rather than per- ver-vertex for each image, sampling must be performed at the hand-level. This makes it challenging tackle class imbalance issue per-hand manner as the class imbalance issue is present in per-vertex. Hence, it is essential to design a principled, hand-level proxy that reflects the underlying per-vertex imbalance. To tackle this challenge, we introduce balanced contact sampling (BCS), which constructs multiple sampling groups in hand-level but still fairly represent both contact and non-contact in per-vertex level. Please refer to the main paper for further details on the BCS.
A.5 More examples of spatial imbalance issue
We visualize the dataset-wide mean contact maps for all dense hand contact datasets used in train-
ing HACO in Figure A2 and Figure A3
Among hand-object interaction datasets, Dex YCB [5],
HO3D [15] H203D 16 ARCTIC
12 HOI4D (36). and H2O 24 exhibit significant spatial
imbalance, with ground tround-truth contact concentrated predominantly on the fingertips. ObMan [19 and
MOW 4 display more promising contact distributions, including high contact probability on the hypothenar region. However, ObMan synthetic dataset with domain gap towards real images, and MOW, while showing diverse palmar contact patterns, lacks dorsal contact and contains fewer than 1,000 samples, making insufficient for large-scale training. Among hand-hand interaction datascts, HIC [59] shows severe spatial imbalance, with contact limited to small fingertip regions and

---

regions between fingers. In contrast, InterHand2.6M 41 offers more favorable contact distribution, covering nearly all hand regions except for small area on the dorsum. Hand-scene scene interaction datasets [1 22 tend to involve large contact areas, primarily on the palmar side. However, interac- tions are often limited to flat surfaces such as ground or walls, resulting low diversity of contact types despite broad coverage. The hand-face interaction dataset, Decaf [55], also exhibits poor spatial diversity due to its limited action types, such as poking or touching the chin. For the hand-body interaction dataset, Hi4D [65] includes extensive contact regions, but most interactions involve full palmar contact during actions such as hugging or patting. As visualized in the dataset-wide mean contact heatmaps, spatial imbalance is prevalent issue across existing dense hand contact datasets. To address this, we propose the vertex- imbalance by reweighting the loss at each vertex based on its mean contact frequency across the dataset. Please refer to the main paper for further details on the VCB loss.
blass-balanced (VCB) loss, which mitigates spatial
Table A2: Comparison with SOTA methods of dense hand contact estimation HIC [59]
RICH (22), Hi4D 665) dataset.
Test dataset
Methods
Precisiont
Recall
FI-Score7
POSA [8]
0.000
0.000
0.000
BSTRO 22
0,000
0.000
0,000
HIC
DECO 58
0.005
0.037
0,006
HACO (Ours)
0.216
0.409
0.263
POSA [8
0.143
0.175
0.125
BSTRO 22
0.437
0.498
0.455
RICH 2
DECO S8
0.324
0.351
0.303
HACO (Ours)
0.741
0.899
0.781
POSA 18
0.128
0.109
0.111
BSTRO 20
0.313
0,207
0.247
Hi4D 65
DECO 58
0.197
0.162
0.146
HACO (Ours)
0.555
0.636
0.565
A.6 More comparisons with state-of-the-art methods
Dense hand contact estimation models. In main manuscript and Table A2 we provided comparison between our HACO and state-of-the-ar (SOTA) methods [8] modification model and its model weights. Nevertheless, the scale of training dataset is largely different between HACO and the SOTA methods. Also, all of the prior methods are dense human- scene contact estimation methods based on human body models such as SMPL (37) and SMPL- X [48] To provide the comparison that excludes the effect of large-scale training and discrepancy of performance rooted by different human models (c.g., SMPL, SMPL-X, MANO). we further provide comparison with SOTA methods that are trained on the same 14 datasets like HACO and modified as hand version in Table To modify the methods to hand version, we replaced all components of SMPL or SMPL-X modules within POSA [8]. BSTRO [22] DECO (58) into MANO (54) modules and changed corresponding model configurations such as the number of vertices accordingly. For POSA, as the method requires estimated 3D mesh, we provide the estimated 3D hand mesh with HaMeR 49 According to Table A3 HACO still consistently outperforms all hand-specific variants of the full-body methods. These results highlight the strength of HACO's task-specific design based on balanced contact sampling and vertex-level class-balanced loss.
22
58
without any
Dense hand contact estimation modules. While not designed as a standalone dense hand contact estimation model, there are several works that present dense hand contact estimation module as part of 3D reconstruction model (55) 21 (5] or pseudo ground-truth generation of dense hand contact [50]. For contact modules in 3D reconstruction model, they can be divided into learning-based
modules 55 61 and heuristic- driven techniques [35 50). These methods were excluded from 21
our SOTA comparison in the main manuscript due to the lack of official code release [15] 61 the task-specific scope of their training data [55] 61], and the heuristic nature of their modules 35 [50]
However, we include a broader evaluation in Table A4 to provide a more comprehensive perspective for future research and to facilitate extensive assessment of hand-specific methods. We first reproduce
DefConNet from Decaf (55), Stage of CHOI (21) and InteractionNet from DICE [6]] based on the model specifications described in their manuscripts, neither the code nor the pretrained weights
25

---

Table A3: Comparison with SOTA methods of dense hand contact estimation with hand version
of POSA 118, BSTRO [22], DECO 58) on MOW [4), HIC [59], RICH [22], Hi4D [65 dataset.
Test dataset
Methods
Precisionf
Recallf
F1-Score1
POSA 18
(hand ver. w/ HaMeR 449 pred. mesh)
0.504
0.400
0,408
BSTRO
(hand ver.)
0,518
0.359
0,380
MOW
DECO
58
(hand ver.)
0.495
0.385
0.395
HACO (Ours)
0.525
0.607
0.522
POSA 18
(hand ver. w/ HaMeR 49 pred. mesh)
0.060
0.225
0.086
BSTRO
(hand ver.)
0.070
0.293
0.107
HIC 59
DECO 58 (hand ver.)
0.067
0,210
0.094
HACO (Ours)
0.216
0.409
0.263
POSA 18 (hand w/ HaMeR 49 pred. mesh)
0.538
0.355
0.398
BSTRO 22 (hand ver.)
0.581
0.413
0,421
RICH 22
DECO [58] (hand ver.)
0.519
0.321
0.367
HACO (Ours)
0.741
0.899
0.781
POSA [8] (hand ver. w/ HaMeR 49 pred. mesh)
0.537
0.257
0.315
BSTRO 22 (hand ver.)
0.594
0.375
0.420
Hi4D 65
DECO 58 (hand ver.)
0.553
0.244
0.304
HACO (Ours)
0.555
0.636
0.565
publicly available. For the Data Engine in LatentAct (50) and EasyHOI we use the officially
released code to generate outputs for evaluation. For all learning-basee modules 15 21 61], we train the reproduced modules based on 14 datasets like HACO but follow their training configurations such as loss function optimizers when provided in their manuscripts. In Table T4ble the results show that our HACO outperforms all previous dense hand contact estimation modules by significant margin. This highlights the need for standalone dense hand contact estimation model specifically trained for hand contact, and demonstrates the effectiveness of HACO's ' spec-fpecific design.
Table A4: Comparison with SOTA methods of dense hand contact estimation with hand contact estimation modules (f: reproduced, re-trained with 14 datasets, code not available) on MOW dataset.
Methods
Precisiont
Recallf
F1-Scoret
DDeCCoNNet (from Decaf [S5]
0.414
0.404
0.368
"Stage in CHOI 22
0.521
0.387
0.407
"InteractionNet (from DICE 61
0.459
0.392
0.413
Data Engine in LatentAct
50
(W/ SAM2
51 & HaMeR 44
0.345
0.244
0.211
EasyHOI [35]
0.480
0.228
0.282
HACO (Ours)
0.525
0.607
0.522
A.7 More ablation studies
Effectiveness of training dataset size. To assess the contribution of large-scale training HACO's performance, we conducted an ablation study comparing different training dataset sizes spanning 3, and 14 datasets. The results demonstrate that performance improves notably as the training dataset size increases, emphasizing the importance of large-scale and diverse data for robust contact estimation. particular, training on the full set using only three representative datasets of Dex YCB [5]. ObMan [19), MOW 4 and even one dataset of MOW dataset still produces competitive results, indicating that HACO remains effective even with limited training dataset size.
datasets yields the best performance, while
A.8 Quantitative results with different backbones
We evaluate the performance of HACO with various backbone architectures. keeping the rest of the pipeline fixed. As shown in A6 our original model using VIT-H (10] pretrained on HaMeR 49) achieves the highest Fc-scoee, demonstrating the strongest overall performance. ViT-based backbones consistently outperform convolutional alternatives, with ViT-B achieving the second highest recall
26

---

Table A5: Ablation for training dataset size MOW 4
dataset.
Methods
Precisiont
Recall
Fl-Scoret
HACO trained on dataset
0.498
0.348
0.373
HACO trained datasets
0.463
0.602
0.485
HACO trained datasets
0.525
0.607
0.522
and ViT-S maintaining strong precision. ViT-L also performs competitively, achieving a strong F1-score. This highlights the effectiveness of Transformer baseed architectures for dense contact estimation, likely due to their global receptive fields and capacity to model long-range dependencies. To assess the benefit of pretraining
related tasks, additionally train and evaluate variant with FPN 31 backbone pretrained with HandOceNet [46], a 3D hand mesh reconstruction model. This configuration yields low F1-score compared to AmageNett-reetrinne backbones, suggesting that reconstructtonnspecifie pretraining alone is insufficient for accurate hand contact prediction. HRNet backbones, widely used in 3D human mesh reconstruction 29
27 achieve moderate performance in our setting, with Fl-scores below 0.5 despite their high-resolution design. ResNet backbones (20) exhibit a wide performance range, with deeper variants such as ResNet-112 performing relatively well, while lighter variants such as ResNet-18 8 offer improved efficiency at the cost of reduced Fl-score. Overall, these results indicate that the performance gains of our final HACO model arise from the combination of the strong representational capacity of the ViT backbone and task-specific knowledge transferred from 3D hand mesh reconstruction.
30
66
Table A6: Comparison of various backbone models on MOW dataset.
Model
Backbone
Pretrained
Precision
Recall
FI-Scoret
HACO
ViT-H 10
HaMeR
44
0.525 0.505
0.607
0.522
HACO
FPN 31
HandOucNet
0.575
0.482
HACO
ViT-L [T0
ImageNet
8
0.493
0.598
0.491
HACO
VIT-B 10
ImageNet
8
0.488
0,604
0.484
HACO
VIT-S 10
ImageNet
0.510
0.514
0.462
HACO
HRNei-Waye
[57
ImageNet
g
0,509
0.563
0.485
HACO
HRNei-W33.
S7
ImageNet
g
0,500
0.558
0.471
HACO
ResNet-152
20
ImageNet
0.506
0.558
0.486
HACO
ResNet- I01
20
ImageNet
0.498
0.564
0.480
HACO
ResNet-50
20
ImageNet
0.490
0.558
0.470
HACO
ResNet-34
20
ImageNet
g
0.489
0.558
0.466
HACO
ResNet-18
20
ImageNet
0.484
0.538
0.456
A.9
Computational requirements
Table A7 reports the computational requirements of HACO with various backbone configurations. The original HACO model, listed in the first row, shows the highest computational cost, particularly in training memory, which exceeds 26GB. This is primarily due to the memory overhead associated with storing gradients while fine-tuning the ViT-H [10] backbone pretrained on HaMeR [4]. To support broader adoption in downstream tasks such 3D hand grasp generation [14 and 3D hand-object
reconstruction [35), we provide additional HACO variants with reduced computational demand.
We first evaluate the FPN [3I] backbone from HandOccNet (46). Although FPN is known to be computationally expensive due to its multi-scale feature hierarchy, it remains significantly more efficient than ViT-H. We also test lighter ViT variants such as ViT-B and ViT-S, which offer favvoralee trade-offs by maintaining moderate memory usage while achieving inference speeds exceeding 60 fps. We further evaluate HRNet [57] backbones, including HRNet- W48 and HRNet- W32. Despite their moderate memory consumption, both models exhibit low inference speeds, falling below 20 fps. Finally, we include ResNet-based (0) variants ranging from lightweight models such as ResNet- 18 to deeper configurations such as ResNet- 152. These models generally show low memory usage, fewer parameters, and high runtime efficiency. In particular, ResNet- 18 achieves an inference speed exceeding 100 fps while maintaining computational cost under 3 GFLOPs.
27

---

Table A7: Computational requirements of various backhone configurations.
Model
Backbone
Train Memory (MB)
Test Memory (MB)
Params. (M)
Speed (fps)
GFLOPs
HACO
ViT-H
63
16,157
12,728
671.10
54.26
125.63
HACO
FPN
6,887
3,002
59.56
66.71
9.94
HACO
ViT-L
63
16,649
4,604
341.91
65.79
62.87
HACO
VIT-B
65
7,767
2,940
122.84
96.80
18.54
HACO
ViT-$
0
4,960
2,434
56.34
86.88
5.10
HACO
HRNet-WARe
W48
57
11,296
7,574
122.37
10.52
23.48
HACO
HRNet-W33 W32
57
9,026
7,806
86.13
17.14
12.55
HACO
ResNet-
152
20
9,252
5,096
103.05
29.84
15.93
HACO
ResNet-101 101
20
7,471
4,978
87.40
41.14
11.07
HACO
ResNet-50
20
5,919
4,826
68.41
73.25
6.21
HACO
ResNet-34
(00
4,163
2,456
56.75
80.85
5.03
HACO
ResNet-18
20
3,785
2,372
46.64
111.23
2.61
A.10
More qualitative results
Additional qualitative comparisons of dense hand contact estimation between HACO and prior methods, including POSA (18) BSTRO (22). and DECO (58), are shown Figures A4 A5 and A6 which correspond to the MOW (4. Hi4D 665. and both HIC [59] and RICH [22 datasets, respectively. Overall, HACO consistently outperforms existing approaches by substantial margin. Unlike prior methods that frequently fail to predict contact
when the hand is clearly interacting, HACO reliably captures accurate dense hand contact. Among the baselines, DECO demonstrates stronger performance than POSA and BSTRO, likely result of being trained on in-the-wild datasets with more diverse contact distributions. However, it remains to be affected by class imbalance, resulting in high rate of false negatives. BSTRO is more severely affected by this imbalance, frequently predicting no contact across the entire hand and producing overly coarse contact regions concentrated the palmar area. This behavior likely stems from its reliance human-scene interaction datasets during training. POSA also exhibits frequent false negatives and fails to detect contact even in visibly interacting regions, likely due its dependence on pose prior rather than direct contact supervision with visual input.
A.11 Limitations and societal impacts
Limitations. Our HACO supports dense hand contact estimation across a wide range of interac- tion types, including hand-object, hand-hand, hand-face, hand-scene, and hand-body interactions. However, hands frequently engage in self-contact under certain poses (e.g., contact between the thumb and index finger an "okay" gesture) or during whole-body motions 442 We intentionally exclude self-contact cases, including them could negatively impact downstream applications. For example, ContactOpt (14) may incorrectly optimize grasps between the hand and an object self-contact is misinterpreted as external contact. As result, HACO currently does not support dense hand contact estimation in self-contact scenarios. Nevertheless, we consider modeling self-contact a promising direction for future research, particularly for applications in the metaverse (e.g., AR/VR) and action recognition. Furthermore, temporal context is essential to learn accurate dense hand contact estimation contact is most noticeable when the movement of hand is obstructed by an that facilitate temporal information from input video stabilize the performance of dense hand contact estimation. With the advent of diffusion-based methods 25 39 designing a diffusion-based approach for dense hand contact estimation is another important research direction as it enables the learning of dense hand contact priors from the large-scale data. Additionally, our balanced contact sampling (BCS) utilizes a single reference point of dataset-wide dense hand contact mean. While this allows HACO to fairly represent diverse contact statistics, if distribution of the statistics is multi-modal this strategy might not be effective. There can be room for improvement on our BCS strategy if the multi-modality of dense hand contact datasets is proven and multi-modal aware methodology can be incorporated to BCS. Moreover, we evaluate HACO only on samples that contain at least one vertex in contact. However, this evaluation protocol may overlook the model's inability to handle false hand-level contact predic- tions, which are important to consider for real-world applications. The incorporation of additional evaluation protocols such accuracy, specificity, and Matthews Correlation Coefficient may be
object or scene. Integration of temporal-basse methods 67 44
28

---

beneficial. Lastly, dense hand contact estimation demonstrates inferior performance on interacting hand scenarios relative other hand interaction scenarios. The incorporation of 3D interacting hand mesh reconstruction method [ 45 or the use of additional training dataset for hand-hand interaction [40] could enhance the performance.
Societal impacts. The proposed method has broad potential for applications involving hand in- teraction analysis, including AR/VR, roboties, and behavioral understanding. However, given the inherently human-centered nature of the task, there is risk of misuse in areas such as mass surveil- lance or privacy infringement. We strongly discourage the deployment of this system in applications that may compromise human rights or personal privacy, and urge practitioners to consider ethical implications when applying HACO to downstream tasks.
29

---

ObMan
DexYCB
MOW
HO3D
H203D
ARCTIC
HOI4D
Figure A2: Dataset-wise dense hand contact mean. These heatmaps show average hand contact of
ground-truths from ObMan 11), YCB [3) MOW [4, HO3D [5] H203D [16), ARCTIC [12]
HOI4D (6) dataset.
30

---

H2O
InterHand2.6M
HIC
PROX
RICH
Decaf
Hi4D
Figure A3: Dataset-wise dense hand contact mean. These heatmaps show average hand contact of
ground-truths from H2O (24), 1M
41 HIC [59], PROX [17). RICH 222 Decaf [55].
Hi4D 65) dataset.
31

---

milfer
Polnar
Danal
Pelssar
Daral
Palnsar
Danal
Palnzz
Input image
POSA
BSTRO
DECO
HACO (Ours)
Figure A4: Qualitative comparison of dense hand contact estimation with POSA [8]
BSTRO 22, DECO [58] MOW [4] dataset. We highlight exemplar regions where HACO outperforms previous methods. Note that we only predict right hand contact.
32

---

Polnar
Danal
Paltsar
Daral
Paltsar
Danal
Palnzz
Input image
POSA
BSTRO
DECO
HACO (Ours)
Figure A5: Qualitative comparison of dense hand contact estimation with POSA [8]
BSTRO 22, DECO (58) Hi4D 65] dataset. We highlight exemplar regions where HACO outperforms previous methods. Note that we only predict right hand contact.
33

---

Polnar
Danal
Peltsar
Darial
Palnsar
Danal
Palnzz
Input image
POSA
BSTRO
DECO
HACO (Ours)
Figure A6: Qualitative comparison of dense hand contact estimation with POSA [8]
BSTRO (22), DECO (58) on HIC (59 and RICH [22 dataset. We highlight exemplar regions where HACO outperforms previous methods. Note that we only predict right hand contact.
34


Reviews:
[{"review_id": "BEvkpicpsk", "reviewer": "Reviewer_a7WB", "summary": "This paper proposes HACO, a framework that recovers dense human contact from visual inputs. As contact labels are intrinsically imbalanced, HACO introduces Balanced Contact Sampling and Vertex-Level Class-Balanced (VCB) loss to mitigate the issue. Balanced Contact Sampling is introduced to mitigate the class imbalance by dividing the dataset into multiple sampling groups that have different levels of hand contact. Vertex-Level Class-Balanced (VCB) loss is built on Class-Balanced loss that uses per-vertex labels for computing the effective number of samples when applying BCE loss on vertex contact labels. With these two components combined, the authors train the contact estimation model across multiple hand datasets based on HaMeR. The experiment results show that HACO's contact estimation results can boost existing hand grasp optimization methods and contact-aware hand-object reconstruction methods.", "questions": "1. (Weakness 1) Is it possible to extend and apply the proposed method to continuous contact representations?", "limitations": "Yes.", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The proposed method targets at direct estimation of dense hand contact from vision inputs, which is an underexplored area.\n2. The proposed method, HACO, is evaluated on actual hand grasp optimization and hand-object reconstruction tasks, where the effectiveness of the estimated hand contact map is verified.\n3. The effectiveness of each component of HACO is validated through extensive ablation studies. In particular, the comparison of the VCB loss with commonly used losses and losses specifically designed for class imbalance demonstrates its effectiveness in addressing vertex-level class imbalance.\n4. This paper is clearly written.\n\nWeaknesses:\n1. (Extension to continuous contact representation) This paper does not contain discussions on how the proposed method can be applied in the regression of continuous contact representation, e.g., offset vectors from hand surface to contact points, or SDF values of hand surface points.", "was_revised": true, "final_justification": "The authors' response resolves my concern about whether the proposed method can be extended to continuous contact representations. The additional experiments in the rebuttal are crucial for supporting the paper's claimed contributions and have to be included in the final manuscript.", "rebuttal": "We thank the reviewer **a7WB** for reviewing our paper.\n\nWe sincerely thank the reviewer for recognizing the strengths of our work. We appreciate the acknowledgment that our method addresses the **underexplored problem of dense hand contact estimation from vision inputs**. We are also glad that the effectiveness of our contact predictions in **hand grasp optimization** and **hand-object reconstruction** tasks was appreciated. Furthermore, we thank the reviewer for highlighting the value of our **extensive ablation studies**, especially the analysis of the **VCB loss**, which demonstrates clear advantages over standard and class imbalance aware losses. Lastly, we are grateful for the comment that the paper is **clearly written**. Below are our responses to the questions mentioned in the review.\n\n### Q: Is it possible to extend and apply the proposed method to continuous contact representations?\n\n**A:** We thank the reviewer for raising this insightful and important question. Our current formulation focuses on discrete (binary) vertex-level contact estimation, which enables interpretability and direct integration with downstream tasks such as contact-driven hand-object optimization and reconstruction. While our representation is discrete, **we recognize that continuous contact fields, such as offset vectors or signed distance functions (SDFs), represent an important and growing research direction in hand contact modeling**.\n\nFrom a training standpoint, both **Balanced Contact Sampling (BCS)** and **Vertex-Level Class-Balanced (VCB) loss** can be extended to regression settings. In the discrete case, BCS mitigates class imbalance between contact and non-contact labels by grouping hand samples based on their overall contact ratio and ensuring that the model sees a more balanced distribution of contact-related and non-contact-related vertices during training. **In the continuous setting**, a similar imbalance arises: **the distribution of contact magnitudes (e.g., SDF values or offset norms) tends to be skewed**, with values indicating distant (non-contact) configurations being much more frequent than those near contact. To address this **distributional imbalance**, BCS can group training samples based on **statistics of the continuous contact signal**, such as the distribution of offset vectors or SDF values from the hand surface, to ensure balanced exposure that does not skew toward either contact or non-contact cases.\n\nThe **VCB loss**, originally designed to address **spatial imbalance** across the hand surface in the discrete case, can also be extended to **continuous contact regression**. In our setting, spatial locations (e.g., mesh vertices or surface regions) exhibit different contact signal distributions. For example, fingertip regions tend to be more frequently in contact, while other areas such as the dorsum are rarely contacted. This imbalance can persist in continuous representations such as SDFs or offset vectors, where certain regions consistently exhibit lower absolute SDF values (i.e., closer proximity), while others predominantly produce large-distance signals. As a result, the regression model may become biased toward frequently occurring signals that is different by spatial region. To address this, VCB loss can be extended by **reweighting the regression loss per spatial location using a continuous weighting strategy** that assigns higher importance to underrepresented contact-relevant signals. This approach generalizes the **vertex-wise effective number of samples strategy** used in our original discrete VCB loss, which discretely re-weights contact and non-contact contributions at each vertex based on their contact frequency. In the continuous case, the weighting function would smoothly vary with the contact signal itself, enabling fine-grained correction of spatial imbalance without relying on discrete labels. This formulation would help the model avoid bias toward dominant signal and encourages learning that is more spatially balanced across the hand surface.\n\nFrom an inference or integration standpoint, HACO’s binary contact predictions can serve as **auxiliary supervision or initialization for continuous contact regression**. For instance, HACO can be used to first localize probable contact regions, allowing a second-stage model to refine predictions into continuous fields such as SDFs or offset vectors. Conversely, a model trained to regress continuous contact fields can have its outputs thresholded or quantized to derive discrete contact labels, which may be useful for downstream tasks.\n\nWe will include this expanded discussion in the final version."}, {"review_id": "MwnXPepdtt", "reviewer": "Reviewer_kFWg", "summary": "This paper proposes HACO, a method for dense hand contact estimation from imbalanced data. While many existing datasets capture various types of hand interactions (e.g., hand-object, hand-hand, hand-scene), they typically suffer from two types of data imbalance: (1) class imbalance, where non-contact samples vastly outnumber actual contact samples, and (2) spatial imbalance, where most hand contact occurs in the fingertip region. To address the class imbalance, HACO employs balanced contact sampling, which ensures fair representation across diverse sample groups. To tackle the spatial imbalance, HACO applies a vertex-level class-balanced loss that reweights the loss contribution of each hand vertex. In experiments, HACO achieves state-of-the-art performance on various hand contact-related tasks, including hand contact estimation, grasp optimization, and joint hand-object reconstruction.", "questions": "Please see the Weaknesses section above.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "[Strengths]\n\n1. Good motivation\n\nAs there is no contact estimation model trained on large-scale, assorted datasets, I agree that pursuing this direction is important for further improving performance. In this regard, addressing the existing datasets' imbalance issues in terms of both class and spatial distribution is a reasonable approach.\n\n2. Writing quality\n\nThe paper is well organized and generally easy to follow. The motivation and technical components are clearly presented.\n\n3. Strong empirical performance\n\nThe proposed method demonstrates strong empirical results across various hand contact-related tasks, including hand contact estimation, 3D hand grasp optimization, and 3D hand-object reconstruction, outperforming existing state-of-the-art methods. The paper also provides comprehensive ablation studies to validate the proposed components, along with extensive qualitative results in the supplementary material.\n\n[Weaknesses]\n\n1. Lack of empirical support for resolving imbalance between contact and non-contact classes\n\nOne of the main motivations of this paper is to address the imbalance between contact and non-contact classes by proposing a method that faithfully learns both. However, the paper states that it “skip[s] fully non-contacting samples during evaluation” (lines 264–265), which seems contradictory. While I understand that existing contact accuracy metrics (e.g., recall, F1-score) are not suitable for evaluating non-contact samples, the paper could at least include an additional binary classification metric (e.g., accuracy in distinguishing contact vs. non-contact) to empirically support the effectiveness of this component. If non-contact samples are entirely discarded during evaluation, then a trivial strategy—such as simply filtering out all non-contact samples, as is done in some interacting hand reconstruction methods—could be a comparably effective alternative, which then questions the motivation for balanced contact sampling.\n\n2. Questionable design for balanced contact sampling\n\nFor balanced contact sampling, the paper groups samples based on a contact balance score, which measures “how much each hand contact sample deviates from the dataset-wide average” (lines 85–86). However, I question whether this can fairly represent the full range of contact cases, particularly if the dataset's ground-truth distribution is multimodal. Moreover, during model training, the use of smoothness regularization encourages predictions to stay close to the average—seemingly contradicting the motivation behind balanced sampling. Clarifying these design choices would be helpful to readers.\n\n3. Lack of justification for spatial imbalance\n\nThe paper claims that existing datasets exhibit spatial imbalance, with contact regions mostly concentrated on the fingertips. However, no evidence is provided to support this claim. As with the class imbalance issue, presenting simple statistics (e.g., contact frequency per vertex) would help validate this point.", "was_revised": true, "final_justification": "My initial questions were well addressed during the author–reviewer discussion. Therefore, I have updated my rating from borderline accept to accept. However, I strongly encourage the authors to clarify in the manuscript all the points discussed in the rebuttal, especially the comparison settings raised by reviewer BqXF.", "rebuttal": "We thank the reviewer **kFWg** for reviewing our paper.\n\nWe sincerely thank the reviewer for highlighting the strengths of our work. We appreciate the recognition of our paper’s **clear motivation**, especially the need for contact estimation models trained on diverse datasets and our approach to **data imbalance**. We are also grateful for the positive remarks on the **clarity of our writing** and the **strong empirical performance** across contact estimation, grasp optimization, and reconstruction tasks, as well as the **comprehensive ablation** and **qualitative results**. Below are our responses to the questions mentioned in the review.\n\n### Q: Lack of empirical support for resolving imbalance between contact and non-contact classes\n**A:** We appreciate the reviewer’s thoughtful comment. Our Balanced Contact Sampling (BCS) is specifically designed to mitigate the class imbalance issue at the **vertex-level**, which is where the contact estimation prediction and supervision occur. Therefore, although we excluded fully non-contacting **hand-level** samples from evaluation due to limitations in existing contact metrics (e.g., recall, F1-score), our evaluation remains **valid within the vertex-level prediction context**, where contact and non-contact vertices co-exist in most samples.\n\nNevertheless, we acknowledge that excluding fully non-contacting hands during evaluation may overlook certain aspects of the contact estimation problem, particularly the model's ability to distinguish between entirely contacting and entirely non-contacting hands at the hand level. To provide a more comprehensive evaluation, we conducted additional experiments (**Table R1**, **Table R2**, **Table R3**) using binary classification metrics of **Accuracy**, **Specificity**, and **MCC (Matthews Correlation Coefficient)**, which are better suited for this scenario. Additionally, since the MOW dataset primarily contains hand-level contacting cases, we evaluate on the RICH dataset with fully non-contacting hand-level samples included, which provides a broader range of contact and non-contact instances.\n\n\n**Table R1. SOTA comparison for hand-level contact estimation on RICH dataset inlcuding fully non-contacting hand samples**\n| Models | Conference   | Accuracy ↑ | Specificity ↑  | MCC ↑  |\n|--------|--------------|-----------|--------------|------|\n| POSA   | CVPR 2021    | 0.561     |  0.285       | 0.222|\n| BSTRO  | CVPR 2022    | 0.627     |\t 0.488       | 0.254|\n| DECO   | ICCV 2023    | 0.614     |\t 0.491       | 0.328|\n| HACO   | -            | **0.802**     |  **0.522**       | **0.604**|\n\n\n**Table R2. SOTA comparison for dense hand contact estimation on RICH dataset inlcuding fully non-contacting hand samples**\n| Models | Conference   | Accuracy ↑  | Specificity ↑  | MCC ↑  |\n|--------|--------------|-----------|--------------|------|\n| POSA   | CVPR 2021    | 0.691     |\t0.768        | 0.392|\n| BSTRO  | CVPR 2022    | 0.839     |\t0.870        | 0.497|\n| DECO   | ICCV 2023    | 0.812     |\t0.905        | 0.557|\n| HACO   | -            | **0.886**     |\t**0.910**        | **0.621**|\n\n\n**Table R3. Ablation study for Balanaced Contact Sampling (BCS) and VCB loss on RICH dataset inlcuding fully non-contacting hand samples**\n| BCS | VCB | Accuracy ↑ | Specificity ↑ | MCC ↑  |\n|-----|-----|----------|-------------|-------|\n| ✗   | ✗   | 0.554    | 0.527       | 0.239 |\n| ✓   | ✗   | 0.608    | 0.616       | 0.323 |\n| ✗   | ✓   | 0.808    | 0.817       | 0.549 |\n| ✓   | ✓   | **0.886**| **0.910**   | **0.621** |\n\nThese results demonstrate that our method remains effective in addressing the contact vs non-contact imbalance, even when fully non-contacting hand-level samples are retained during evaluation.\n\nRegarding the concern about a trivial strategy such as **filtering out all non-contacting samples** during training, we emphasize that this approach is **not applicable in our setting**. **Our method explicitly targets vertex-level** class imbalance, where **hand-level** contact is **irrelevant** in the **training standpoint**. Therefore, we **do not need to discard entire hand-level** non-contacting samples. In fact, doing so would **reduce the diversity of non-contact patterns seen during training**, which is counterproductive for learning a **robust vertex-level contact estimator**. By retaining all samples and addressing imbalance at the vertex-level through Balanced Contact Sampling and VCB loss, our method is able to learn from both contact and non-contact regions in a fine-grained manner, enabling more robust and accurate contact prediction.\n\nWe will revise the final version to better clarify this design choice and include evaluation results with fully non-contacting hand samples.\n\n\n### Q: Questionable design for balanced contact sampling\n**A:** Thank you for the insightful question. Our goal with **Balanced Contact Sampling (BCS)** is to mitigate the **severe imbalance in vertex-level contact labels**, a common challenge in real-world datasets where **most hand vertices are non-contact**. Since the **ground-truth contact labels are provided per-hand** (rather than per vertex-image pair), sampling must be performed at the hand-level, making it **essential to design a principled, hand-level proxy that reflects the underlying vertex-level imbalance**.\n\nTo achieve this, we compute a **dataset-wide average contact vector as a neutral reference** and define a per-hand contact balance score based on the **deviation of each hand’s contact vector from this mean**. This scalar score provides a consistent and interpretable way to **stratify** hands by how contact-heavy, contact-sparse, or atypical their vertex-level contact patterns are. The mean is not used to enforce a unimodal assumption, but to **offer a common reference point** across thousands of heterogeneous contact configurations.\n\nWe emphasize that BCS is **not designed to simply span the extremes of the contact spectrum** (e.g., from fully contact to fully non-contact), but rather to increase exposure to diverse and underrepresented hand contact patterns. Real-world datasets tend to be skewed and sparse along this spectrum, making **naive or uniform sampling approaches** risk **over-representing common patterns** or producing **arbitrary diversity without structure**. In contrast, **BCS** applies non-linear binning and stratified sampling based on the contact balance score to construct a training subset that is **structurally balanced and representative**.\n\nHowever, we believe that future extensions of HACO could incorporate clustering or interaction-aware multi-modality modeling if the presence of such multi-modality is well analyzed. We will clarify these points and the implications of our sampling strategy in the final version.\n\nRegarding the concern about **regularization**, we acknowledge that the regularization term encourages predictions to not deviate excessively from the dataset-wide contact mean. While this may seem to contrast with the goal of BCS, which promotes diverse contact patterns, **these two components serve different but complementary purposes**. The regularization acts as a stabilizer to **prevent overfitting to extreme or noisy contact configurations**, ensuring that the model remains within a reasonable solution space. This is similar to many standard regularization strategies in deep learning, including adversarial training or entropy regularization, which temper the main loss without undermining its core objective. In our case, BCS enhances the diversity of training signals, while the regularization ensures stability and robustness.\n\nWe will revise the final version to clarify these design choices and explicitly address this point.\n\n\n### Q: Lack of justification for spatial imbalance\n**A:** Thank you for the thoughtful feedback. We visualize spatial imbalance using heatmaps on the hand surface in **Figure 1-b** of the main manuscript for **dataset-wide imbalance** and **Figures A2 and A3** in the supplementary material for **per-dataset imbalance**. These visualizations highlight the spatial imbalance present for all 14 datasets. In particular, they show that **fingertips tend to exhibit higher contact probability** (in red) compared to other regions of the hand in many datasets. \n\nHowever, we agree that including **raw data statistics**, such as bar plots showing the contact to non contact ratio per vertex, would offer a more detailed and quantitative view of the spatial imbalance. Unfortunately, due to the limitations of the rebuttal format, we were unable to include these visualizations at this stage. We will incorporate this additional analysis in the final version to enhance clarity and depth."}, {"review_id": "LWyESyXgf5", "reviewer": "Reviewer_BqXF", "summary": "The paper describes a method, HACO, that addresses two problems of imbalanced data during hand interactions: class imbalance (due to the vast majority of non-contact frames) and spatial imbalance (due to the prevalence of fingertips contacts). The first problem is tackled by resampling the datasets based on contact balance scores to ensure fair representation. Spatial imbalance is instead tackled by reweighting vertices to give more importance to the under-represented ones. These two strategies lead to improvements in contact estimation and 3D hand reconstruction during HOI.", "questions": "1.\tCan the authors justify their evaluation choices? In particular:\na. Table 4: the non-logical ordering starting with non-target domains (e.g., MOW starting with HS instead of HO)\nb. Table 5: comparing HACO with full-body contact methods (POSA, BSTRO, DECO) on HO tasks. The goal of addressing class imbalance and spatial imbalance is related but not the same as out-of-distribution generalization.\nc. the reason for using MOW for primary evaluation despite acknowledging GT problems in supplementary material\n2.\tCan the authors ablate each of the two components of HACO while keeping the other one (BCS-only vs. VCB-only vs. both)?", "limitations": "The authors addressed limitations and potential negative societal impact in the supplementary material. However, the limitations section could be greatly expanded to critically discuss failure cases that could open up new research avenues. The current section only mentions that they excluded self-contact.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Quality\n\nStrengths:\n•\tAddresses a challenging and important problem of data imbalance in dense hand contact estimation with two complementary solutions.\n•\tProposes technically sound approaches: Balanced Contact Sampling (BCS) for class imbalance and Vertex-Level Class-Balanced (VCB) loss for spatial imbalance.\n\nWeaknesses:\n•\tEvaluation choices: Fundamentally flawed baseline comparisons where POSA, BSTRO, DECO are designed for full-body HS but evaluated on HO tasks; primary evaluation on MOW dataset despite authors acknowledging GT problems in supplementary material; only fair comparisons (DeepContact, EasyHOI) show minimal improvements.\n•\tMethodological gaps: Table 4 ablation ordering is not motivated and it starts with non-target domains; unclear contribution of HACO vs. massive 14-dataset training.\n•\tMissing critical ablation: Tables 2&3 show identical baseline performance (0.525, 0.632, 0.531), suggesting BCS and VCB were not properly isolated.\n•\tTechnical inconsistencies: Fragmented contact predictions contradict claimed \"smoothness loss\" benefits mentioned in L238-239.\n•\tGround truth reliability concerns: Missing GT visualizations in Figure 3 make it impossible to assess prediction quality; some predictions appear inconsistent with visual evidence (e.g., in row 3, based on the shadow, fingers do not seem to touch the computer).\n\n\nClarity\n\nStrengths:\n•\tClear problem formulation identifying class imbalance and spatial imbalance as distinct challenges.\n•\tCodes will be released.\n\n\nWeaknesses:\n•\tSection 3.2 (Balanced Contact Sampling) clarity can be improved. \n•\tMisleading baseline comparisons: Paper does not justify why quantitatively evaluating POSA, BSTRO, DECO on MOW as they solve fundamentally different problems.\n•\tInconsistent categorization: Not clear why Decaf (hand-face) and Hi4D (whole-body) are categorized together in hand-body despite being fundamentally different.\n•\tIncomplete experimental details: Table 4 ordering rationale unexplained, no explanation on why using MOW for evaluation as it has the worst GT estimation or why using out-of-domain baselines.\n\n\nSignificance\n\nStrengths:\n•\tTackles fundamental challenge in hand interaction understanding that affects multiple downstream applications.\n\nWeaknesses:\n•\tLimited practical validation: Only meaningful comparisons (DeepContact, EasyHOI) show modest improvements.\n•\tInflated performance claims: Dramatic improvements in Table 5 come from comparing incompatible methods.\n•\tUnclear method vs. data contribution: Cannot assess whether improvements stem from HACO techniques or from combining 14 diverse datasets.\n•\tEvaluation reliability: Primary evaluation on acknowledged problematic dataset (MOW) undermines confidence in reported improvements.\n•\tFailure case analysis: Missing discussion of when/why the method fails or performs poorly.\n\n\nOriginality\n\nStrengths:\n•\tAdaptation of class-balanced loss to vertex-level to address different aspects of data imbalance.\n•\tLarge-scale multi-domain training with resampling approach for hand contact estimation.", "was_revised": true, "final_justification": "I would like to thank the authors for addressing all my comments. Given the new experiments and the changes they have promised to incorporate into the manuscript, I raised my evaluation.", "rebuttal": "We thank the reviewer **BqXF** for reviewing our paper.\n\nWe appreciate the recognition of our method’s contributions, including the use of **Balanced Contact Sampling (BCS)** and **Vertex-Level Class-Balanced (VCB) loss** to address data imbalance in dense hand contact estimation. We are also grateful for highlighting the **clarity & originality of our formulation**, the **planned code release**, and the **significance** of our work for hand interaction tasks. Below are our responses to the questions mentioned in the review.\n\n### Q: Flawed baseline comparisons where POSA, BSTRO, DECO are designed for full-body HS but evaluated on HO tasks\n**A:**  We thank the reviewer for this important concern. Our goal in comparing HACO with full-body contact models (POSA, BSTRO, DECO) was to **demonstrate HACO’s capability in the broader context of hand contact estimation**, a task **lacking dedicated baselines**.\n\nIn real-world use cases, **full-body models (e.g., DECO) are often the only available tools** for hand contact estimation from a single RGB image. To address this practical gap, we compared HACO with these methods.\n\nHowever, we acknowledge the need for broader evaluation. Hence, **we implement and compare hand-specific versions** of POSA, BSTRO, and DECO by replacing their SMPL/SMPL-X modules with MANO and training them on the same 14 datasets as HACO (**Table R1**). Additionally, **we reproduced hand contact modules** from DefConNet (Decaf), Stage 1 in CHOI, and InteractionNet (DICE) (**Table R2**). While not designed solely for dense hand contact task, these methods offer relevant baselines.\n\n**Table R1. SOTA comparison for dense hand contact estimation with hand version of POSA, BSTRO, DECO on MOW dataset**\n| Model                 | Conference  | Precision | Recall | F1-Score |\n|-----------------------|-------------|-----------|--------|----------|\n| POSA (hand ver. w/ HaMeR pred mesh) | CVPR 2021  | 0.504     | 0.400  | 0.408    |\n| BSTRO (hand ver.)         | CVPR 2022   | 0.518     | 0.359  | 0.380    |\n| DECO (hand ver.)          | ICCV 2023   | 0.495     | 0.385  | 0.395    |\n| **HACO (Ours)**      | -              | **0.525** | **0.632** | **0.531** |\n\n\n**Table R2. SOTA comparison for dense hand contact estimation with hand contact estimation modules (†: reproduced, re-trained with 14 datasets, code not available) on MOW dataset**\n| Model         | Conference         | Precision | Recall | F1-Score |\n|---------------|--------------------|-----------|--------|----------|\n| †DefConNet (from Decaf)     | SIGGRAPH Asia 2023 | 0.414         | 0.404      | 0.368        |\n| †Stage 1 in CHOI          | AAAI 2024          | 0.521     | 0.387  | 0.407    |\n| †InteractionNet (from DICE [A])| ICLR 2025          | 0.459     | 0.392  | 0.413    |\n| Data Engine in LatentAct [B] (w/ SAM2 [C] & HaMeR)     | CVPR 2025          | 0.345     | 0.244  | 0.211    |\n| EasyHOI       | CVPR 2025          | 0.480     | 0.228  | 0.282    |\n| **HACO (Ours)** | -                | **0.525** | **0.632** | **0.531** |\n\nHACO consistently outperforms both hand-specific variants of full-body methods and dedicated hand contact modules. These results highlight the strength of HACO’s task-specific design. We will include them in the final version to provide a more comprehensive evaluation.\n\n\n### Q: Primary evaluation on MOW dataset despite GT problems\n**A:**  We thank the reviewer for this important observation. While we acknowledge that the MOW dataset still contains spatial imbalance, it is a key benchmark with **relatively lower spatial imbalance** compared to other HO datasets (e.g., DexYCB, HOI4D) and **still serves as important baseline** for HOI tasks (e.g., HORT [D]).\n\nWe selected MOW as our primary benchmark as it is **closest to real-world hand contact scenarios** among the 14 datasets. In contrast, most other datasets are collected in controlled settings, making them less suitable for evaluating dense contact under realistic, unconstrained conditions.\n\nNevertheless, we agree that including additional evaluation provides a broader perspective. To address this, **we conducted additional SOTA comparison on the HIC, RICH, and Hi4D dataset** as part of response for reviewer **xnje** in its **Table R2, R3, and R4**.\n\nAs shown in the results, HACO continues to outperform prior methods by a substantial margin. We will include these results in the final version.\n\n\n### Q: DeepContact, EasyHOI experiments show minimal improvements\n**A:**  We have added experiments in **Table R1 and R2** to facilitate broader comparisons with significant improvements. We will add them to our final version.\n\n\n### Q: Table 4 ablation ordering is not motivated and it starts with non-target domains\n**A:**  The purpose of **Table 4** in main manuscript is to show that **diverse hand interaction datasets are critical** for robust contact estimation. To highlight the **impact of dataset diversity**, we intentionally **excluded the target interaction category for the first three rows** (e.g., exclude HO when evaluating on MOW, or HS when evaluating on RICH).\n\nThe table is ordered to **begin with the most generalizable interaction types** (e.g., hand-object, hand-scene), which provide broader contact coverage than narrower categories like hand-hand. Starting with the same category as the evaluation set could lead to overfitting and would obscure the benefit of diverse training.\n\nBy progressively adding categories, we emphasize that diversity is key to accurate and robust dense contact estimation.\n\n\n### Q: Unclear contribution of HACO vs. massive 14-dataset training\n**A:**  We thank the reviewer for raising this point. To assess the contribution of large-scale training to HACO’s performance, we conducted an ablation study comparing different training dataset size. The results below show that while training on the full set of 14 datasets improves performance, **HACO still achieves competitive results when trained on only 3 datasets** (DexYCB, ObMan, MOW), demonstrating that the method’s effectiveness does not solely rely on large-scale data.\n\n**Table R3. Ablation for training dataset size on MOW dataset**\n| Models | Precision | Recall | F1-Score |\n|--------|-----------|--------|----------|\n| HACO trained on 3 datasets   | 0.463 | 0.602 | 0.485 |\n| HACO trained on 14 datasets  | **0.525** | **0.632** | **0.531** |\n\nWe will include this ablation study in the final version of the paper to better clarify the individual contributions of the model design and training data scale.\n\n\n### Q: In Tables 2 & 3, BCS and VCB were not properly isolated\n**A:**  We thank the reviewer for pointing out this concern. Our ablation experiments in **Tables 2 and 3** in main manuscript were intended to show the individual contributions of Balanced Contact Sampling (BCS) and Vertex-Level Class-Balanced (VCB) loss relative to the full HACO model, following standard ablation protocols adopted in many previous works. However, we acknowledge that the original tables did not include a baseline where both BCS and VCB were removed, which is helpful to fully isolate their effects.\n\nTo address this, **we provide an extended ablation study** in **Table R4**, which includes all combinations of BCS and VCB. This allows for a clearer interpretation of each component’s contribution and their combined effect on performance.\n\n**Table R4. Ablation for Balanced Contact Sampling and VCB loss on MOW dataset**\n| BCS  | VCB  | Precision | Recall | F1-Score |\n|------|------|-----------|--------|----------|\n| ✗   | ✗   | 0.503     | 0.242  | 0.310    |\n| ✓   | ✗   | **0.530**     | 0.294  | 0.348    |\n| ✗   | ✓   | 0.520     | 0.542  | 0.481    |\n| ✓   | ✓   | 0.525   | **0.632**  | **0.531**    |\n\nWe will include this full ablation in the final version of the paper to better clarify the individual and joint contributions of BCS and VCB loss.\n\n\n### Q. Fragmented contact predictions contradict claimed \"smoothness loss\" benefits mentioned in L238-239\n**A:**  Fragmented contact refers to spatially isolated predictions. As noted in L238–239, our smoothness loss penalizes such discontinuities, promoting more coherent and anatomically plausible results. We will clarify this in the final version.\n\n### Q. Missing GT visualizations in Figure 3\n**A:**  Thank you for your thoughtful feedback. We will include GT visualizations in the final version to improve interpretability.\n\n### Q. Some predictions inconsistent with visual evidence\n**A:**  We acknowledge that HACO, as the first dense hand contact estimator trained on large-scale datasets, still has limitations. We will provide a detailed discussion of failure cases to guide future improvements.\n\n### Q. Section 3.2 clarity can be improved\n**A:**  Thank you for your feedback. We will revise Section 3.2 to improve clarity on Balanced Contact Sampling.\n\n### Q. Unclear grouping of Decaf (hand-face) and Hi4D (whole-body)\n**A:**  Thank you for your detailed feedback. We will strictly categorize Decaf and Hi4D in different interaction category and fix any mis-categorization accordingly throughout the entire manuscript.\n\n### Q. Inflated performance claims: Dramatic improvements in Table 5 come from comparing incompatible methods\n**A:**  We now include stronger and more broader baselines that is re-trained on the same dataset with HACO. We will add them to final manuscript.\n\n### Q. Failure case analysis: Missing discussion of when/why the method fails or performs poorly\n**A:**  We will include detailed failure cases and limitations to help future researchers build upon HACO.\n\n[A] Wu et al. DICE: End-to-end Deformation Capture of Hand-Face Interactions from a Single Image. In ICLR, 2025.\n\n[B] Prakash et al. How Do I Do That? Synthesizing 3D Hand Motion and Contacts for Everyday Interactions. In CVPR, 2025.\n\n[C] Ravi et al. SAM 2: Segment Anything in Images and Videos. In ICLR, 2025.\n\n[D] Chen et al. HORT: Monocular Hand-held Objects Reconstruction with Transformers. In ArXiv, 2025."}, {"review_id": "CVaeaQBPY5", "reviewer": "Reviewer_xnje", "summary": "This paper develops the HACO framework to address the class imbalance and spatial imbalance issues in hand contact datasets. HACO tackles class imbalance with balanced contact sampling and spatial imbalance with a vertex-level class-balanced (VCB) loss. Experiments compared with the SoTA methods show the effectiveness of the proposed method.", "questions": "See cons above.", "limitations": "yes", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Pros:\n1. The paper is well-written and easy to follow.\n2. The ablation study is comprehensive and makes the paper convincing.\n\nCons:\n1. Limited novelty: The proposed Balanced contact sampling and Vertex-level class balanced loss is straightforward. Balanced contact sampling is simply resampling all training data from datasets to make data balanced. The class balanced loss is frequently used to handle imbalanced training issue and the authors simply apply it on vertex level which I think is not enough to be a solid technical contribution. \n2. Since the proposed method is to solve the imbalanced data issue, I think experiments on the data with different classes might be helpful to verify the effectiveness of the method. For example, previous methods may perform well on the whole dataset but do not work well on another kind of data which is a minor class in the datasets. \n3. Can you further explain the rationale of network structure design? Why self-sttention+cross attention? Why adding contact token? Is it learnable? \n4. I found that the comparison with SoTA methods is only conducted on MOW dataset. Testing them on more other datasets will be better.\n5. Can you give more details about how to compare with DeepContact since your model only outputs contact map?", "was_revised": true, "final_justification": "I appreciate the authors' responses, which addressed some of my concerns. However, I still believe the technical contribution of the work is limited, as the proposed Balanced Contact Sampling and Vertex-level Class Balanced Loss are straightforward and lack novelty. Additionally, the unfair comparison pointed out by reviewer BqXF raises further doubts about the reliability of the experimental results. Therefore, I maintain my initial rating.", "rebuttal": "We thank the reviewer **xnje** for reviewing our paper.\n\nWe sincerely appreciate the reviewer’s positive assessments of our work. The recognition that the paper is **clearly written and accessible** is especially encouraging. We also thank the reviewer for highlighting the **comprehensiveness of our ablation studies**, which we believe strengthens the credibility of our contributions. Below, we address the specific points raised in the review.\n\n\n### Q: Limited novelty of the proposed methods\n**A:** While **Balanced Contact Sampling (BCS)** and **VCB loss** build on known principles, both are carefully adapted to address the **specific challenges of dense hand contact estimation**, which requires **precise vertex-level prediction under severe class and spatial imbalance**, common in real-world hand interaction datasets.\n\n**BCS is not a simple resampling heuristic**, but a **dataset-driven strategy** targeting **class imbalance**. It groups training samples by a **contact balance score**, exposing the model to both contact-sparse and contact-heavy hands. Since **most hand vertices are non-contact**, BCS is essential to avoid overfitting to dominant patterns of non-contact. As it uses **dataset-level statistics**, it remains effective across varying datasets.\n\n**VCB loss** addresses **spatial imbalance** by applying **vertex-specific reweighting** based on contact frequency. This prevents overfitting to frequently contacted regions (e.g., fingertips) and promotes learning from underrepresented regions. **Unlike generic balancing techniques**, VCB directly resolves the **spatial imbalance nature of dense hand contact datasets** and is broadly applicable through its reliance on label statistics.\n\nOur method yields strong gains over prior work and supports downstream tasks like 3D grasp optimization and hand-object reconstruction. By explicitly addressing class and spatial imbalance, it provides both **practical benefits and conceptual contributions** to dense hand contact estimation.\n\nWe will clarify these motivations and contributions more explicitly in the final version.\n\n\n### Q: Since the proposed method is to solve the imbalanced data issue, I think experiments on the data with different classes might be helpful to verify the effectiveness of the method\n**A:** We appreciate the reviewer’s insightful suggestion. To **directly evaluate the effectiveness of our method under imbalance issue**, we constructed a **test set composed of highly imbalanced samples**. Specifically, we selected the top 500 samples with the **highest contact balance scores** (as defined in Eq. 1 of the main paper) from a **combined pool of MOW, HIC, RICH, and Hi4D datasets**. This subset explicitly reflects extreme imbalance, making it well-suited to assess the robustness of imbalance-aware methods.\n\n**Table R1. SOTA comparison for dense hand contact estimation on assorted dataset with high imbalance**\n| Models | Conference  | Precision | Recall | F1-Score |\n|--------|-------------|-----------|--------|----------|\n| POSA   | CVPR 2021   | 0.117 | 0.102 | 0.104 |\n| BSTRO  | CVPR 2022   | 0.243 | 0.188 | 0.204 |\n| DECO   | ICCV 2023   | 0.225 | 0.205 | 0.192 |\n| HACO (Ours)   | -    | **0.550** | **0.626** | **0.576** |\n\nAs shown above, HACO achieves the best performance across all metrics, demonstrating strong robustness even on samples with severe imbalance. This **directly confirms the impact of our method in handling imbalance issue**. We will include these results and the process of dataset acquisition in the final version.\n\n\n### Q: Can you further explain the rationale of network structure design? Why self-sttention+cross attention? Why adding contact token? Is it learnable?\n**A:** Our model architecture closely follows the HaMeR framework, which has demonstrated strong performance in hand mesh recovery task. This design decision reflects our focus not on proposing a novel network architecture, but rather on advancing contact estimation through: (1) training on a large-scale composite dataset spanning 14 independent datasets, (2) Balanced Contact Sampling (BCS) to address class imbalance, and (3) Vertex-Level Class-Balanced (VCB) loss to address spatial imbalance in hand contact.\n\nRegarding the self-attention and cross-attention modules, **we adopted the structure used in the officially released HaMeR implementation**. This combination has proven effective for fusing image features with mesh queries, and we retained it to ensure fair comparison and isolate the effects of our proposed training strategies.\n\nThe **contact token** used in our architecture serves as a **query token** for cross-attention Transformer (as mentioned in L154), which is also analogous to the query token used in HaMeR's official implementation. Similar to the class token in Vision Transformers, this **learnable contact token** serves as a **global summary vector** for predicting vertex-level contact. By learning this token end-to-end, the model is encouraged to develop a contact-specific representation that guides the Transformer’s attention during **feature aggregation**. This design is **widely adopted in Transformer-based frameworks** where there needs effective feature aggregation for regression or classification task.\n\nWe will clarify these architectural choices more explicitly in the final version.\n\n\n\n### Q: I found that the comparison with SoTA methods is only conducted on MOW dataset. Testing them on more other datasets will be better.\n**A:** We appreciate the reviewer’s suggestion and agree that **broader evaluation across multiple datasets** provides a more comprehensive assessment. In response, we conducted additional comparisons on three more benchmark datasets: **HIC**, **RICH**, and **Hi4D**. These datasets represent diverse hand interaction scenarios and include **hand-hand**, **hand-scene**, and **hand-body contacts**.\n\n**Table R2. SOTA comparison for dense hand contact estimation on HIC dataset**\n| Models | Conference  | Precision | Recall | F1-Score |\n|--------|-------------|-----------|--------|----------|\n| POSA   | CVPR 2021   | 0.000     | 0.000  | 0.000    |\n| BSTRO  | CVPR 2022   | 0.000     | 0.000  | 0.000    |\n| DECO   | ICCV 2023   | 0.005     | 0.037  | 0.006    |\n| HACO (Ours)   | -           | **0.181**     | **0.641**  | **0.278**    |\n\n**Table R3. SOTA comparison for dense hand contact estimation on RICH dataset**\n| Models | Conference  | Precision | Recall | F1-Score |\n|--------|-------------|-----------|--------|----------|\n| POSA   | CVPR 2021   | 0.143     | 0.175  | 0.125    |\n| BSTRO  | CVPR 2022   | 0.437     | 0.498  | 0.455    |\n| DECO   | ICCV 2023   | 0.324     | 0.351  | 0.303    |\n| HACO (Ours)   | -           | **0.743**     | **0.570**  | **0.528**    |\n\n\n**Table R4. SOTA comparison for dense hand contact estimation on Hi4D dataset**\n| Models | Conference  | Precision | Recall | F1-Score |\n|--------|-------------|-----------|--------|----------|\n| POSA   | CVPR 2021   | 0.128     | 0.109  |\t0.111    |\n| BSTRO  | CVPR 2022   | 0.313     | 0.207  |\t0.247    |\n| DECO   | ICCV 2023   | 0.197     | 0.162  | 0.146    |\n| HACO (Ours)   | -           | **0.781**     | **0.923**  | **0.828**    |\n\nAcross all three additional datasets, HACO significantly outperforms prior SOTA methods, confirming its robustness and accuracy in diverse interaction settings. These additional results will be included in the final version of the manuscript.\n\n\n### Q: Can you give more details about how to compare with DeepContact since your model only outputs contact map?\n**A:** Our comparison with DeepContact is conducted using the **ContactOpt framework**, which optimizes coarse 3D hand and object predictions using contact information. **We utilize ContactOpt’s Differentiable Contact Optimization stage** to refine the coarse 3D hand and object poses (from HFL-Net) for both DeepContact and HACO experiment.\n\n**The key difference lies in how contact is estimated**: **DeepContact** predicts contact from the **coarse 3D hand and object meshes** (extracted using HFL-Net), while **HACO** predicts contact **directly from the RGB image**. Note that DeepContact does not use image information in any manner. Rather, it relies solely on the geometry of the coarse 3D hand and object meshes to estimate contact. As **3D hand grasp optimization inevitably requires initial 3D hand and object poses**, we use the same HFL-Net predictions for both methods during ContactOpt’s Differentiable Contact Optimization stage. In HACO’s case, we supply the HACO's estimated contact from image alongside these coarse 3D poses to ContactOpt’s optimization process.\n\nAs shown in Section 5.4 and Table 6 in main paper, HACO achieves **comparable or superior results across various metrics**. These results demonstrate that **HACO's contact prediction enables robust and effective contact-guided 3D grasp optimization**, despite relying only on image input during contact prediction.\n\nWe will clarify this protocol more explicitly in the final version."}]

Author rebuttal:

Meta-review: (a) The paper proposed a dense hand‑contact estimation method by explicitly addressing two forms of imbalance common in today’s multi‑dataset settings: 1) class imbalance (much more non‑contact than contact) via Balanced Contact Sampling (BCS); 2) spatial imbalance (contacts concentrated at fingertips) via a vertex‑level class‑balanced (VCB) loss. The authors also evaluate downstream impact on grasp optimization (ContactOpt) and hand–object reconstruction. 

(b) + The problem is well‑scoped, which is an underexplored task (dense contact), with principled handling of class/spatial imbalance and clear downstream impact (grasp optimization, reconstruction).
    +  Paper is well-written and organized. 

(c) - BCS score design could be further stress‑tested for multimodality (beyond mean‑deviation).
     - Should ensure promised GT overlays and failure analysis, and make it into the camera‑ready.

(d)(e) The paper got mixed rating {5, 5,4,3}. The rebuttal materially addressed fairness, metrics with non‑contact hands, and isolation of method vs data‑scale—moving the balance to borderline‑positive. While not architecturally novel, the paper makes a useful, evidence‑backed contribution to a task where imbalance hurts both learning and downstream usage. AC agreed with the acceptance decision.
Decision: Accept (poster)