SiMHand: Mining Similar Hands for Large-Scale 3D Hand Pose Pre-training

Authors: Nie Lin, Takehiko Ohkawa, Yifei Huang, Mingfang Zhang, Minjie Cai, Ming Li, Ryosuke Furuta, Yoichi Sato
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: 3D Hand Pose Estimation; Contrastive Learning; Pre-Training of Large-Scale Images;

Abstract:
We present a framework for pre-training of 3D hand pose estimation from in-the-wild hand images sharing with similar hand characteristics, dubbed SiMHand. Pre-training with large-scale images achieves promising results in various tasks, but prior methods for 3D hand pose pre-training have not fully utilized the potential of diverse hand images accessible from in-the-wild videos. To facilitate scalable pre-training, we first prepare an extensive pool of hand images from in-the-wild videos and design our pre-training method with contrastive learning. Specifically, we collect over 2.0M hand images from recent human-centric videos, such as 100DOH and Ego4D. To extract discriminative information from these images, we focus on the similarity of hands: pairs of non-identical samples with similar hand poses. We then propose a novel contrastive learning method that embeds similar hand pairs closer in the feature space. Our method not only learns from similar samples but also adaptively weights the contrastive learning loss based on inter-sample distance, leading to additional performance gains. Our experiments demonstrate that our method outperforms conventional contrastive learning approaches that produce positive pairs solely from a single image with data augmentation. We achieve significant improvements over the state-of-the-art method (PeCLR) in various datasets, with gains of 15% on FreiHand, 10% on DexYCB, and 4% on AssemblyHands. Our code is available at https://github.com/ut-vision/SiMHand.

Decision: accept (poster)
Reviewer scores: 6, 6, 8

Reviews and discussion:
<thread>
creation date:
1730371412065

thread_type:
official_review

<thread_object>
creator: reviewer_pnqd

<content>
summary:
This paper addresses the task of self-supervised learning for 3D hand pose estimation from monocular RGB. The authors build on prior work in the area and improve upon it in three main areas: 1) Use of noisy 2D supervision to mine positive samples 2) Adaptive weighting that weighs positive and negative samples based off of their distance of the 2D keypoints 3) Processing and use of Ego4D and 100DOH for self-supervision.
Their proposed method first constructs a pose embedding based off of the noisy acquired 2D keypoints using an off-the-shelf predictor. This pose embedding is then used to mine positive samples given a query image.
These positive samples are used within the contrastive loss as positive samples, whereas the remaining image in the batch is marked as negative. The positive and negative samples are weighted additionally using weights that are computed based off of the scaled euclidean distance of their 2D keypoints.
The self-supervised model is trained on Ego4D and/or 100DOH. Those datasets have been processed using an off-the-shelf hand detector model. Supervised training was done on a variety of supervised datasets. Experimental results show large improvement across all benchmark datasets compared to prior self-supervised models.

strengths:
- The paper empirically verifies the improvement over prior self-supervised models. Self-supervision is a rather underexplored area in hand pose estimation and can lead to potentially great benefit as foundation models.
- The improvements are substantial
- The paper is easy to understand

weaknesses:
- The method shows great improvement over prior self-supervised methods through the use of noisy 2D annotations. However, its use is a rather involved process: it first needs to be embedded, then used during pre-training before then performing supervised fine-tuning. Instead, why not just use the noisy 2D annotations directly as a form of weak-supervision? In fact, this has been done in prior work [1] and has lead to substantial improvements. In order to properly verify the usefulness of the authors proposed method, there first needs to be a baseline showcasing that the straightforward addition of the noisy 2D annotation during pre-training or supervised training performs comparatively worse. Otherwise why should one employ the authors proposed method? Due to my own experiences in the field, I fear that the weak-supervision approach will outperform the authors proposed approach.
- The paper does not compare to other related work in the field for which test results on FreiHand, DexYCB and AssemblyHands are available. Without these, we cannot assess properly the value of this work and how it fits in overall.


[1] Weakly-Supervised Mesh-Convolutional Hand Reconstruction in the Wild, Kulon et al., CVPR'20

questions:
- How does this work compare to a weakly-supervised approach with noisy annotations?
- How does this work perform compared to other related work on the tested datasets?
- Instead of using weights, could one instead not use a more appropriate loss that will automatically lead to larger effects depending on the samples weights? For example, MSE will automatically weight the contributions of more distant samples stronger.
- L143-144: Why balance the number of left and right hand if they all end up being converted to right-handed images?
- Eq 1: Why not use the cosine similarity which is more popular for distances in feature space?
- Fig3: The colored boxes at the end of the model pipeline seems to be in the wrong order. E.g the figure shows positive samples minimizing alignment.
-L238-239: rough -> noisy
- Table 1: What is "baseline"? This needs to be explained in the image caption
- Table 1: Why are the worst result of SimCLR in bold? Shouldn't the most performant number be in bold?
- Not all figures and tables are referred to in text.
- Table 3: inconsistent capitalization of simclr etc. This also occurs occasionally in the text.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730448807410

thread_type:
official_review

<thread_object>
creator: reviewer_4kod

<content>
summary:
- The paper explores pretraining for hand pose estimation using a large number of 2D image samples.
- It introduces HandCLR, a contrastive learning-based method. This method expands the definition of positive and negative samples by using pairs with similar actions from different sources, improving upon previous methods that relied solely on data augmentation.
- The authors collected extensive pretraining datasets from 100DOH and Ego4D and studied effective methods for mining similar hand samples.
- They designed a Top-K sampling strategy for positive and negative samples and implemented adaptive weighting.
- Experiments show that the proposed method outperforms baselines in both pretraining and downstream finetuning tasks.

strengths:
- The paper is well-written and easy to follow.
- The motivation behind the proposed method is sound, with comprehensive details from data preparation to training.
- The design of contrastive loss with weighting provides better gradient guidance for samples with different sources and similarities, which is both reasonable and effective.
- The numerous experiments reflect significant effort by the authors.
- The experimental section is logical and thorough, demonstrating performance improvements across different datasets and analyzing the impact of training samples, finetuning sample size, and various design modules.

weaknesses:
- Some presentation issues need improvement
    - Figure 6 should be updated to remove inappropriate "bbox" spelling marks. Additionally, all images in the paper should be replaced with vector versions to prevent blurry text, as seen in Figure 3.
- The article lacks references and discussions on self-supervised methods. The recent two works, S2Hand and HaMuCo, although not pre-training methods, also attempt to use unlabeled images and 2D off-the-shelf detectors to train 3D hand pose estimation models.
- Otherwise, the paper is relatively complete with no major weaknesses 

[1] Model-based 3D Hand Reconstruction via Self-Supervised Learning

[2] HaMuCo: Hand Pose Estimation via Multiview Collaborative Self-Supervised Learning

questions:
- Why are the baseline metrics relatively poor? For example, Freihand dataset shows 18+ MPJPE, while recent works (i.e. MobRecon) often achieve <6 PA-MPJPE. Could you explain if Procrustes analysis accounts for such a large performance difference? If the author could explicitly address this performance gap or more clearly explain the difference between the baseline metrics and those of existing fully supervised methods, it would be better.
- Are the positive sample augmentations identical to those used for query images?
- Is Figure 4 showing results from the FreiHand dataset?
- Regarding minibatch construction, the authors mention using 2N samples (N query images and their corresponding positive samples). Using the top-1 method for defining positive samples, could there be cases where a negative sample $I_n$ for query image $I_m$ is actually very similar but not top-1 (e.g., top-K where K>1)? Do the authors have more detailed descriptions of how to increase the discrimination in positive/negative sample sampling, or is it solely addressed through adaptive weighting?
- How is diversity ensured in the reverse lookup of top-1 samples for each query image? Could there be cases where samples from videos j and k are mutually top-1 similar samples, potentially reducing training diversity by constantly pairing samples from the same two videos?
- What specific models were used as baselines in Tab.1?

- Since the baselines compared by the author all have open-source code, in order to enhance the reproducibility of the article and the usability for downstream tasks, it is hoped that the author will adhere to what is mentioned in the article and actually release the code
</content>
</thread_object>

</thread>
<thread>
creation date:
1730860637321

thread_type:
official_review

<thread_object>
creator: reviewer_vbov

<content>
summary:
This paper presents a contrastive learning method for the pre-training of 3D hand pose estimation based on large-scale in-the-wild hand images. A parameter-free adaptive weighting mechanism is introduced in the contrastive learning loss, which not only learns from similar samples but also adaptively weights the contrastive learning loss based on inter-sample distance. Experiments show improved performance compared with existing pre-training methods.

strengths:
- The paper is well written and easy to follow.
- The motivation of finding similar hands derived from different video domains is technically sound, which can further benefit contrastive learning process from discriminating foreground hands in varying backgrounds.
- The experimental results in Table 3 demonstrate the generality of the proposed contrastive learning with adaptive weighting mechanism.

weaknesses:
- TempCLR [1] proposes a pre-train framework for 3D hand reconstruction with time-coherent contrastive learning, and shows better performance compared with PeCLR. Although TempCLR focuses on reconstruction tasks, the used parametric model can output 3D pose results. Therefore, more comparisons with TempCLR would be helpful.

- In the second column of Figure 6, HandCLR demonstrates advanced performance in hand-object occlusion. Does the proposed method exhibit robustness in similar severe occlusion scenarios involving hand-object interactions? More qualitative analysis in datasets like DexYCB or in-the-wild scenarios would be helpful.

- The proposed adaptive weighting mechanism is a straightforward approach that has proven effective; however, it lacks a clear articulation of its motivation, particularly regarding the challenges faced in 3D hand pose estimation tasks as mentioned in the introduction.

[1] Tempclr: Reconstructing hands via time-coherent contrastive learning. In  3DV, 2022.

questions:
N/A
</content>
</thread_object>

</thread>
<thread>
creation date:
1732962972970

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global response (Summary of the rebuttal period)

comment:
**We would like to thank the area chairs and reviewers for their efforts and suggestions in reviewing our paper**.  We have revised the manuscript according to the reviewers' suggestions and **marked all important modifications in blue** in the new version of the submission. **The important differences include** (W: Weakness; Q: Question):
* **Motivation Clarifications**: Following Reviewer vbov (R1)'s Weakness 3 (W3), we have updated the section on "adaptive weighting" in the introduction section to address the comment regarding the "lack of a clear expression of motivation."
* **Detailed Related Work**: Following Reviewer 4Kod (R2)'s Weakness 2 (W2), we have updated the related work section by incorporating discussions of two relevant studies: S2Hand and HaMuCo, thus enhancing the discussion on self-supervised methods.
* **More Visualization**: Following Reviewer vbov (R1)'s Weakness 2 (W2), we have revised the visualization part in the experiment section by adding a discussion on Hand-Object occlusion and providing visual results on the DexYCB dataset.

As minor revision, the experimental text descriptions have been accordingly updated based on the reviewers' suggestions and subtly adjusted to align with these modifications: 
* **R2-Q6, R3-Q7, R3-Q8**: Changed “baseline” to “w/o pre-training” in Tab.1 and Tab.2 for clarity, and improved result notation by using bold for the best and underlining for the second-best results. Captions were updated accordingly.
* **R2-Q3**: Revised Fig.4's caption to include the FreiHand dataset.

Additionally, **we have provided the requested experimental results** in response to the reviewers' concerns about Weaknesses & Questions **(R1-W1, R3-Q1,2,3)**. **During the rebuttal period, the additional experiments we included were as follows**:
* **TempCLR Comparison**: To address Reviewer vbov's (R1) Weakness 1 (W1), we provided a comparison of TempCLR under different pre-training scales. Additionally, we explained the challenges we encountered when applying TempCLR to large-scale wild hand data.
* **WSL Setting Comparison**: To address Reviewer PnQd's (R3) Question 1 (Q1), we provided comparison results in the weakly supervised setting. We also explained the differences between the weakly supervised learning (WSL) setup and pre-training.
* **3D HPE Method Comparison**: To address Reviewer PnQd's (R3) Question 2 (Q2), we provided a comparison of the dataset with other 3D hand pose estimation (HPE) methods.
* **MSE Loss Comparison**: In response to Reviewer PnQd's (R3) Question 3 (Q3), we provided the results of a model pre-trained using MSE loss and subsequently fine-tuned it.

**We will release the data, code, and pre-processed assets (e.g., hand bounding boxes, frame indices from Ego4D and 100DOH, 2D keypoints, and similar hand labels) corresponding to 2 million (2.0M) large-scale wild hand images to promote the development of the research community.**

In conclusion, we sincerely thank the area chairs and reviewers again for their valuable feedback, which has significantly improved the quality of our work.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a contrastive-learning method for pre-training of 3D hand pose estimation based on large-scale in-the-wild data. 

The three reviewers all appreciate the straightforward nature and effectiveness of the approach, especially as it is demonstrated on large-scale data.

Some weaknesses raised include issues of presentation, as well as discussion and placement of the proposed method with respect to the larger body of literature on self-supervised learning.  This has largely been addressed through the author response.  

The AC has read through the reviewer comments and author responses.  Given the strong support by all the reviewers (6,6,8), the AC recommends that the paper be accepted.  The authors are requested to incorporate the content in their response to reviewers to their camera-ready version of the paper.