Title: Learning Dense Hand Contact Estimation from Imbalanced Data
Venue: NeurIPS 2025
Accepted: yes
Keywords: Hand contact estimation

Hand-object interaction
Abstract: Hands are essential to human interaction, and exploring contact between hands and the world can promote comprehensive understanding of their function. Recently, there have been growing number of hand interaction datasets that cover interaction with object, other hand, scene, and body. Despite the significance of the task and increasing high-quality data, how to effectively learn dense hand contact estimation remains largely underexplored. There are two major challenges for learning dense hand contact estimation. First, there exists class imbalance issue from hand contact datasets where majority of regions are not in contact. Second, hand contact datasets contain spatial imbalance issue with most of hand contact exhibited in finger tips, resulting in challenges for generalization towards contacts in other hand regions. To tackle these issues, we present a framework that learns dense HAnd COntact estimation (HACO) from imbalanced data. To resolve the class imbalance issue, we introduce balanced contact sampling, which builds and samples from multiple sampling groups that fairly represent diverse contact statistics for both contact and non-contact vertices. Moreover, to address the spatial imbalance issue, we propose vertex-level class-balanced (VCB) loss, which incorporates spatially varying contact distribution by separately reweighting loss contribution of each vertex based on its contact frequency across dataset. As a result, we effectively learn to predict dense hand contact estimation with large-scale hand contact data without suffering from class and spatial imbalance issue. The codes are available at https://github.com/dqj5182/HACO_RELEASE.


Peer reviews:
comments: SUMMARY:
This paper develops the HACO framework to address the class imbalance and spatial imbalance issues in hand contact datasets. HACO tackles class imbalance with balanced contact sampling and spatial imbalance with a vertex-level class-balanced (VCB) loss. Experiments compared with the SoTA methods show the effectiveness of the proposed method.

STRENGTHS AND WEAKNESSES:
Pros:
1. The paper is well-written and easy to follow.
2. The ablation study is comprehensive and makes the paper convincing.

Cons:
1. Limited novelty: The proposed Balanced contact sampling and Vertex-level class balanced loss is straightforward. Balanced contact sampling is simply resampling all training data from datasets to make data balanced. The class balanced loss is frequently used to handle imbalanced training issue and the authors simply apply it on vertex level which I think is not enough to be a solid technical contribution. 
2. Since the proposed method is to solve the imbalanced data issue, I think experiments on the data with different classes might be helpful to verify the effectiveness of the method. For example, previous methods may perform well on the whole dataset but do not work well on another kind of data which is a minor class in the datasets. 
3. Can you further explain the rationale of network structure design? Why self-sttention+cross attention? Why adding contact token? Is it learnable? 
4. I found that the comparison with SoTA methods is only conducted on MOW dataset. Testing them on more other datasets will be better.
5. Can you give more details about how to compare with DeepContact since your model only outputs contact map?

QUESTIONS:
See cons above.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-06-25

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper describes a method, HACO, that addresses two problems of imbalanced data during hand interactions: class imbalance (due to the vast majority of non-contact frames) and spatial imbalance (due to the prevalence of fingertips contacts). The first problem is tackled by resampling the datasets based on contact balance scores to ensure fair representation. Spatial imbalance is instead tackled by reweighting vertices to give more importance to the under-represented ones. These two strategies lead to improvements in contact estimation and 3D hand reconstruction during HOI.\n\nSTRENGTHS AND WEAKNESSES:\nQuality\n\nStrengths:\n•\tAddresses a challenging and important problem of data imbalance in dense hand contact estimation with two complementary solutions.\n•\tProposes technically sound approaches: Balanced Contact Sampling (BCS) for class imbalance and Vertex-Level Class-Balanced (VCB) loss for spatial imbalance.\n\nWeaknesses:\n•\tEvaluation choices: Fundamentally flawed baseline comparisons where POSA, BSTRO, DECO are designed for full-body HS but evaluated on HO tasks; primary evaluation on MOW dataset despite authors acknowledging GT problems in supplementary material; only fair comparisons (DeepContact, EasyHOI) show minimal improvements.\n•\tMethodological gaps: Table 4 ablation ordering is not motivated and it starts with non-target domains; unclear contribution of HACO vs. massive 14-dataset training.\n•\tMissing critical ablation: Tables 2&3 show identical baseline performance (0.525, 0.632, 0.531), suggesting BCS and VCB were not properly isolated.\n•\tTechnical inconsistencies: Fragmented contact predictions contradict claimed \"smoothness loss\" benefits mentioned in L238-239.\n•\tGround truth reliability concerns: Missing GT visualizations in Figure 3 make it impossible to assess prediction quality; some predictions appear inconsistent with visual evidence (e.g., in row 3, based on the shadow, fingers do not seem to touch the computer).\n\n\nClarity\n\nStrengths:\n•\tClear problem formulation identifying class imbalance and spatial imbalance as distinct challenges.\n•\tCodes will be released.\n\n\nWeaknesses:\n•\tSection 3.2 (Balanced Contact Sampling) clarity can be improved. \n•\tMisleading baseline comparisons: Paper does not justify why quantitatively evaluating POSA, BSTRO, DECO on MOW as they solve fundamentally different problems.\n•\tInconsistent categorization: Not clear why Decaf (hand-face) and Hi4D (whole-body) are categorized together in hand-body despite being fundamentally different.\n•\tIncomplete experimental details: Table 4 ordering rationale unexplained, no explanation on why using MOW for evaluation as it has the worst GT estimation or why using out-of-domain baselines.\n\n\nSignificance\n\nStrengths:\n•\tTackles fundamental challenge in hand interaction understanding that affects multiple downstream applications.\n\nWeaknesses:\n•\tLimited practical validation: Only meaningful comparisons (DeepContact, EasyHOI) show modest improvements.\n•\tInflated performance claims: Dramatic improvements in Table 5 come from comparing incompatible methods.\n•\tUnclear method vs. data contribution: Cannot assess whether improvements stem from HACO techniques or from combining 14 diverse datasets.\n•\tEvaluation reliability: Primary evaluation on acknowledged problematic dataset (MOW) undermines confidence in reported improvements.\n•\tFailure case analysis: Missing discussion of when/why the method fails or performs poorly.\n\n\nOriginality\n\nStrengths:\n•\tAdaptation of class-balanced loss to vertex-level to address different aspects of data imbalance.\n•\tLarge-scale multi-domain training with resampling approach for hand contact estimation.\n\nQUESTIONS:\n1.\tCan the authors justify their evaluation choices? In particular:\na. Table 4: the non-logical ordering starting with non-target domains (e.g., MOW starting with HS instead of HO)\nb. Table 5: comparing HACO with full-body contact methods (POSA, BSTRO, DECO) on HO tasks. The goal of addressing class imbalance and spatial imbalance is related but not the same as out-of-distribution generalization.\nc. the reason for using MOW for primary evaluation despite acknowledging GT problems in supplementary material\n2.\tCan the authors ablate each of the two components of HACO while keeping the other one (BCS-only vs. VCB-only vs. both)?\n\nLIMITATIONS:\nThe authors addressed limitations and potential negative societal impact in the supplementary material. However, the limitations section could be greatly expanded to critically discuss failure cases that could open up new research avenues. The current section only mentions that they excluded self-contact.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes HACO, a method for dense hand contact estimation from imbalanced data. While many existing datasets capture various types of hand interactions (e.g., hand-object, hand-hand, hand-scene), they typically suffer from two types of data imbalance: (1) class imbalance, where non-contact samples vastly outnumber actual contact samples, and (2) spatial imbalance, where most hand contact occurs in the fingertip region. To address the class imbalance, HACO employs balanced contact sampling, which ensures fair representation across diverse sample groups. To tackle the spatial imbalance, HACO applies a vertex-level class-balanced loss that reweights the loss contribution of each hand vertex. In experiments, HACO achieves state-of-the-art performance on various hand contact-related tasks, including hand contact estimation, grasp optimization, and joint hand-object reconstruction.\n\nSTRENGTHS AND WEAKNESSES:\n[Strengths]\n\n1. Good motivation\n\nAs there is no contact estimation model trained on large-scale, assorted datasets, I agree that pursuing this direction is important for further improving performance. In this regard, addressing the existing datasets' imbalance issues in terms of both class and spatial distribution is a reasonable approach.\n\n2. Writing quality\n\nThe paper is well organized and generally easy to follow. The motivation and technical components are clearly presented.\n\n3. Strong empirical performance\n\nThe proposed method demonstrates strong empirical results across various hand contact-related tasks, including hand contact estimation, 3D hand grasp optimization, and 3D hand-object reconstruction, outperforming existing state-of-the-art methods. The paper also provides comprehensive ablation studies to validate the proposed components, along with extensive qualitative results in the supplementary material.\n\n[Weaknesses]\n\n1. Lack of empirical support for resolving imbalance between contact and non-contact classes\n\nOne of the main motivations of this paper is to address the imbalance between contact and non-contact classes by proposing a method that faithfully learns both. However, the paper states that it “skip[s] fully non-contacting samples during evaluation” (lines 264–265), which seems contradictory. While I understand that existing contact accuracy metrics (e.g., recall, F1-score) are not suitable for evaluating non-contact samples, the paper could at least include an additional binary classification metric (e.g., accuracy in distinguishing contact vs. non-contact) to empirically support the effectiveness of this component. If non-contact samples are entirely discarded during evaluation, then a trivial strategy—such as simply filtering out all non-contact samples, as is done in some interacting hand reconstruction methods—could be a comparably effective alternative, which then questions the motivation for balanced contact sampling.\n\n2. Questionable design for balanced contact sampling\n\nFor balanced contact sampling, the paper groups samples based on a contact balance score, which measures “how much each hand contact sample deviates from the dataset-wide average” (lines 85–86). However, I question whether this can fairly represent the full range of contact cases, particularly if the dataset's ground-truth distribution is multimodal. Moreover, during model training, the use of smoothness regularization encourages predictions to stay close to the average—seemingly contradicting the motivation behind balanced sampling. Clarifying these design choices would be helpful to readers.\n\n3. Lack of justification for spatial imbalance\n\nThe paper claims that existing datasets exhibit spatial imbalance, with contact regions mostly concentrated on the fingertips. However, no evidence is provided to support this claim. As with the class imbalance issue, presenting simple statistics (e.g., contact frequency per vertex) would help validate this point.\n\nQUESTIONS:\nPlease see the Weaknesses section above.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes HACO, a framework that recovers dense human contact from visual inputs. As contact labels are intrinsically imbalanced, HACO introduces Balanced Contact Sampling and Vertex-Level Class-Balanced (VCB) loss to mitigate the issue. Balanced Contact Sampling is introduced to mitigate the class imbalance by dividing the dataset into multiple sampling groups that have different levels of hand contact. Vertex-Level Class-Balanced (VCB) loss is built on Class-Balanced loss that uses per-vertex labels for computing the effective number of samples when applying BCE loss on vertex contact labels. With these two components combined, the authors train the contact estimation model across multiple hand datasets based on HaMeR. The experiment results show that HACO's contact estimation results can boost existing hand grasp optimization methods and contact-aware hand-object reconstruction methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The proposed method targets at direct estimation of dense hand contact from vision inputs, which is an underexplored area.\n2. The proposed method, HACO, is evaluated on actual hand grasp optimization and hand-object reconstruction tasks, where the effectiveness of the estimated hand contact map is verified.\n3. The effectiveness of each component of HACO is validated through extensive ablation studies. In particular, the comparison of the VCB loss with commonly used losses and losses specifically designed for class imbalance demonstrates its effectiveness in addressing vertex-level class imbalance.\n4. This paper is clearly written.\n\nWeaknesses:\n1. (Extension to continuous contact representation) This paper does not contain discussions on how the proposed method can be applied in the regression of continuous contact representation, e.g., offset vectors from hand surface to contact points, or SDF values of hand surface points.\n\nQUESTIONS:\n1. (Weakness 1) Is it possible to extend and apply the proposed method to continuous contact representations?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-01"}