SiMHand: Mining Similar Hands for Large-Scale 3D Hand Pose Pre-training
Authors: Nie Lin
Takehiko Ohkawa
Yifei Huang
Mingfang Zhang
Minjie Cai
Ming Li
Ryosuke Furuta
Yoichi Sato
Venue: ICLR 2025 Poster (2025)
Keywords: 3D Hand Pose Estimation; Contrastive Learning; Pre-Training of Large-Scale Images;
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: We present a framework for pre-training of 3D hand pose estimation from in-the-wild hand images sharing with similar hand characteristics, dubbed SiMHand. Pre-training with large-scale images achieves promising results in various tasks, but prior methods for 3D hand pose pre-training have not fully utilized the potential of diverse hand images accessible from in-the-wild videos. To facilitate scalable pre-training, we first prepare an extensive pool of hand images from in-the-wild videos and design our pre-training method with contrastive learning. Specifically, we collect over 2.0M hand images from recent human-centric videos, such as 100DOH and Ego4D. To extract discriminative information from these images, we focus on the similarity of hands: pairs of non-identical samples with similar hand poses. We then propose a novel contrastive learning method that embeds similar hand pairs closer in the feature space. Our method not only learns from similar samples but also adaptively weights the contrastive learning loss based on inter-sample distance, leading to additional performance gains. Our experiments demonstrate that our method outperforms conventional contrastive learning approaches that produce positive pairs solely from a single image with data augmentation. We achieve significant improvements over the state-of-the-art method (PeCLR) in various datasets, with gains of 15% on FreiHand, 10% on DexYCB, and 4% on AssemblyHands. Our code is available at https://github.com/ut-vision/SiMHand.

Published as a conference paper at ICLR 2025
SIMHAND: MINING SIMILAR HANDS FOR LARGE- SCALE 3D HAND POSE PRE-TRAINING
Nie Lin Takehiko Ohkawa' ' Yifei Huang
Mingfang Zhang', Minjie Caf Ming Li',
Ryosuke Furuta' & Yoichi Sato!
'The University of Tokyo, HHuman University
{nielin, ohkawa-t, hyf, mfzhang mingli, furuta, ysato) uut.ki..utoky..aee jp. caiminjie ieehnu. edu. en, 1i 1i-ming9488g ecc. wer.cc.uutokyoe u-tokyo. ac. jo
ABSTRACT
We present a framework for pre-training of 3D hand pose estimation from in- the-wild hand images sharing with similar hand characteristics, dubbed SiM- Hand. Pre-training with large-scale images achieves promising results in vari- tasks, but prior methods for 3D hand pose pre-training have not fully uti- lized the potential of diverse hand images accessible from in-the-wild videos. To facilitate scalable pre-training, we first prepare extensive pool of hand im- ages from in-the-wild videos and design our pre- trainingi method with contrastive learning. Specifically, we collect over 2.0M hand images from recent human- centric videos, such as 100DOH and Ego4D. To extract discriminative informa- tion from these images, we focus on the similarity of hands: pairs of non-identical samples with similar hand poses. We then propose novel contrastive learning method that embeds similar hand pairs closer in the feature space. Our method not only learns from similar samples but also adaptively weights the contrastive learning loss based on inter-sample distance, leading to additional performance gains. Our experiments demonstrate that method outperforms conventional contrastive learning approaches that produce positive pairs solely from a single image with data augmentation. We achieve significant improvements over the state-of-the-ard method (PeCLR) in various datasets, with gains of 15% on Frei- Hand, 10% DexYCB, and 4% on AssemblyHands. Our code is available vttps://gitnne com ut vision/ SiMHand
INTRODUCTION
Hands serve trigger for interact with the world, as seen in various human-centric videos.
The precise tracking of hand states, such as 3D keypoints, is crucial for video understanding (Sener
iet al.) 2022 Wen al. 2023), AR/VR interfaces (Han et al.] 2022 Wu et al. 2020). and robot learning (Chao et al. |021] n et al. 2022). To this end, 3D hand pose estimation has been studied
through constructing labeled datasets (Ohkawa et al. 2023a Zimmermann al. 2019 Chao al.
2021
Ohkawa et al. 2023b) and advancing supervised pose estimators (Cai et al.) 2018 Ge et al, Liu al. 2024 Fan al., 2024). However, utilizing large-scale unannotated hand videos for pre-training remains underexplored, while collections of human-centri vidcos. like
2019
Park et [2.22
[022]]
3,670 hours of videos from Ego4D (Grauman et al. 2022 and day videos from 100DOH (Shan
et al. 2020), are readily available.
In pre-training, contrastive learning has been utilized to learn from unlabeled images like Sim- CLR (Chen et al. (2020), which maximizes agreement between positive pairs while repelling neg- atives. Spurr el al. (Spurr et al. (00T)) introduce pose equivariant contrastive learning (PeCLR) for 3D hand pose estimation, which aligns the geometry of features encoded from augmented im- ages with affine transformations. However, both SimCLR and PeCLR create positive pairs from a single sample by applying data augmentation, limiting the gains from positive pairs their hand appearance and backgrounds are identical. Ziani er al. (Ziani et al., 2022) extend the contrastive learning framework to video sequences by treating temporally adjacent hand crops positive pairs.
Equal contribution. Corresponding author.

---

Published as a conference paper at ICLR 2025
Previous methods
Same sample with ougmenstions
SIMHand (Ours)
Pre-training
Pre-tralming
Encoder
hildd
Projcton
Zina
Curensstvve
Fine-tuning
Freet
Downdtioa
Similar hands from different Images
Large-soale hands
characleriatics
Different
[
Basiground
3D Hard Poon
Apperemento
Figure 1: The pipeline of pre-training and fine-tuning, (Left) Previous pre- pre-training methods (e.g., PeCLR (Spurr et al.] (021T) learn from positive pairs originating from the different augmentations and tue-tune the network on a dataset. (Right) Our method designed to learn from positive pairs with similar foreground hands, sampled from pool of hand images in the wild.
However, in- will videos can challenge tracking hands across frames, especially in egocentric views where hands are often unobservable due to camera motion. Meanwhile, this temporal positive sample mining remains the limited appearance variation of hands and backgrounds.
In this work, we introduce SiMHand, novel contrastive learning framework for 3D hand pose pre-training, which leverages diverse hand images in the wild, with the largest 3D hand pose pre- training set to date. We specifically collect 2.0M hand images from human-centric videos, from Ego4D (Grauman et al. 2022) and 100DOH (Shan al. 2020), using an off-the-shelff hand de- tector (Shan et al.] (2020). Our pre-training set significantly exceeds the scale of prior works by two orders of magnitude, such as over 32-47K images in (Spurr et al. 202I) and 86K images from 100DOH in (Ziani et al. 2022).
Our method focuses on learning discriminative information by mining hands with similar charac- teristics from various video domains. Based benefit from discriminating the foreground of hands in varying backgrounds. As shown in Fig. our positive pairs are sourced from different images, offering additional information gains from different types of object interactions, backgrounds, and hand appearances. Specifically, we use an off- shffthe-shelf 2D hand pose estimator (Lugaresi al. 2019) to identify similar hands from the pre- training set.
observations, contrastive learning further
Using the identified similar hands as positive pairs, we further propose adaptive weighting, to dy- namically find informative pairs during training. A naive adaptation of the similar hands is to replace the original positive pairs in contrastive learning, but this scheme struggles to exploit how similar the paired hands are. To tackle this, we assign weights based on the similarity scores within the mini-batch the contrastive learning loss. The weights are designed to have higher values as the similarity of the pairs increases. This allows the optimization of contrastive learning to explicitly consider the proximity of samples, beyond binary discrimination between positives and negatives.
We validate the effectiveness of the pre-trained networks by fine-tuning several datasets for 3D hand pose estimation, namely FreiHand (Zimmermann et al. 2019), DexYCB (Chao et al. 2021), and AssemblyHande (Ohkawa et al. 2023b). Our proposed method consistently outperforms ventional contrastive learning methods, SimCLR and PeCLR. Additionally, we conduct extensive ablation experiments to analyze: 1) performance with varying pre-training and fine-tuning data sizes, 2) the effect of adaptive weighting, and 3) the improvement with different levels of similarity.
In summary, the main contribution of this paper is threefold:
We propose SiMHand, a contrastive learning method for 3D hand pose pre- pre-training, lever- aging positive samples with similar hands mined from 2.0M in-the-wild hand images.
We introduce a parameter-free adaptive weighting mechanism the contrastive learning loss, enabling optimization guidance according to the calculated similarity.
Our experiments demonstrate that our approach surpasses prior pre- pre-training methods and achieves robust performances across different hand pose datasets.
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORK
The task of 3D hand pose estimation aims to regress 3D hand joints. Since annotating 3D hand poses is challenging, only limited labeled datasets are available (Ohkawal let al. 2023a), and most of which are constructed in controlled laboratory settings (Zimmermann jet al. 2019; Chao et al.) 2021; Moon et al. 2020; Ohkawa 2hkaaa et al. (023b). Given this challenge, two approaches have been proposed to facilitate learning from limited annotations: pseudo-labeling and self-supervised pre-training. Pseudo-labeling methods learn from pseudo-ground-rre
3D hand pose estimation:
assigned on
2021 Yang et al. 2021 Ohkawa it al. 2022; Liu et al.) 2024). For example, S2Hand (Chen et al.
unlabeled images (Chen al. 2021c; Zheng et al. 2023 Liu et al.
2021c attempls to learn 3D pose
only from noisy 2D keypoints on single-view image, while HaMuCo (Zheng al., 2023 extends
such self- self-supervised learning to multi-view setups. Alternatively, pre-training methods aim to find well-initialized models with unlabeled data for downstream tasks. Prior works propose contrastive learning approaches but rely on relatively small pre-training sets (e.g., 32-47K images in (Spurr et al. 202T) and 86K images in (Ziani et al. p.222). We collect hand images from large human-
centrie datasets such as Ego4D (Grauman al. 2022 and 100DOH (Shan et al. 2020), expanding our pre- pre-training set to 2.0M images.
Contrastive learning has emerged as powerful technique in self-supervised learning, bringing positive samples closer while pushing negative samples apart (Chopra et al. 2005 SScrroff et al.) 2013 Song et al.] [0166] [Sohn] 2016 He al.) [2020) [Huang al. (023)
Contrastive learning:
Standard methods generate positive samples from an identical image with data augmentation (i.e..
self- self-positives) (Grill al. 2020 Caron al., 2020; Chen & He| 2021| Radford al. 2021| Caron
iet al. (021). thus the positive supervision doesn't I explicitly model inter-sample relationships. To address this, Zhang et al. propose a relaxed extension of self-positives,
con-self-positivee (Zhang it al. 2022). which share similar characteristics but originate different images, such as images cap- turing the same scene (Arandjelovic et al. 2016 Ge et al.) 2020 Berton al., 2022 Hausler al.] 2021]. the same person ID (Chen et al. 2021a b), and miew images (Jie et al. 2024). The positive supervision from non-self- -positive enables considering diverse inter-sample a alignment and facilitates the learning of semantics more easily. Zhang el al. identify non-self-positives by search- ing similar human skeletons from single-view images and adapt in action recognition (Zhang et al.) 2022). Jie et al. rely multi-view (i.e. paired) images to define con-sslf-positivee and propose pair- wise weights to adaptively leverage useful multi-view pairs @Jie et al.) 2024). Our work proposes the mining of con-self-positivee from 2D keypoint cues with additional pair-wise weighting to account for similarity from unpaired data in pre-training.
METHOD
Our approach SiMHand aims to pre-train an encoder for 3D hand pose estimation with large-scale human-centric videos in the wild. We first construct pre-training from egocentric and exocentric hand videos (Sec. p. Then, we find similar hand images to define positive pairs across videos (Sec. [3.2). Finally, incorporate these positive pairs into contrastive learning framework and employ adaptive weights to improve the effectiveness in pre-training (Sec. 3.3).
3.1 DATA PREPROCESSING
Our preprocessing involves creating a set of valid hand images for pre-training, which is sampled from a set of N videos: (11, (2,, .., J. select valid frames with visible hands. Given a video frame I E v, the model detects the existence of the hand and its bounding box, creating hand crops enclosing either hand identity (right/left) from Trnn- To avoid bias related to hand identity, we balance the number of right and left hand crops equally and then convert all crops to right-hand images. Then, we create a set of frames for each video v; as F, = 44.2 I,,,, where 4i, E H x3 represents the processed crop with height H and width W, and T; is the total number of crops in v. The height H and width W are defined post-resize to give the uniform image size. Using this frame set F,, the video dataset can be re-represented
We use an off-the-shelf hand detector (Shan et 2020) to
V F2.... Fx). Specifically, we processed two datasets, Ego4D (Grauman et al.) 2022) and 100DOH (Shan et al.) 2020). to collect 1.0M images from 8K and 21 K videos, respectively. More details about our preprocessing can be found in the supplement.
3

---

Published as a conference paper at ICLR 2025
Neighbor frame
Top-2
Top-5
Top-10
Top-50
Top-1
Top-100
Top-500
Top-1000
Top-5000
Figure 2: Visualization similar hand samples in Top-K. Given the query image (/), the mined similar samples are shown ("Top-1" corresponds to I in Sec. 3.2
3.2 MINING SIMILAR HANDS
To incorporate diverse samples in contrastive learning, we design positive pairs from non-identical images with similar foreground hands. Here we construct a mining algorithm to find similar hands from V by focusing on pose similarity between hand images. We first extract 2D keypoints from I, embed in the feature space, and search positive sample.
Pose embedding: We adopt estimated 2D keypoints (for 21 joints) find similar hands. We use an off off-the-shelf 2D hand pose estimator o (Lugaresi et al., 2019). but the outputs are prone to be noisy testing the wild. To make it more robust, we obtain D-dimensional embedding of 2D hand keypoints, p E RD, for each image 7. This serves reduce the noise effect while preserving the semantics hands. We use a concatenated 42-dimensional vector as the output of á for later use. Particularly, we apply PCA-based dimension reduction, which projects the keypoints vector into a fower-dimensionas space of size D. Given the PCA projection matrix M E R-2XD, the pose embedding calculated p M² o(1).
Mining: This step is designed to identify a positive sample I+ RRHxXx3 x3 paired with a query image I. We denote the similarity mining logic I+ SiM(I). As shown in Fig. 2 using the closest (neighbor) sample in the PCA space encounters a trivial solution I E Us where both images originate from the same video ve. Similarly to (Ziani al., (022), the supervision by neighbor samples of the same video has less diversity in backgrounds, hand appearances, and object interactions. Thus Specifically, we search the minimum distance within the set of all frames except for e, written as Fi Un Fs. Given an query Itjj which represents the j-th image of the i-th video, the function SiM(.) formulated as
motivated to find similar hands derived from different videos.
SiM(Ii,) arg min ef D(M 6(x), (T...))
where D(:,.) is the Euclidean distance metric.
As proof of concept, we illustrate examples after our mining SiM(-) in Fig. 2 We denote "Top-1" (most similar)
assigned positive sample I to the query image I. As references, the rest of the figures ("Top-K"") represent the K-th similar samples. Our sampling highlights the diversity in captured environments and interactions, while it also suggests that the rank (distance) increases, the sampled images become dissimilar. Additional visualization results of similar hands can be found in supplement.
3.3
CONTRASTIVE LEARNING FROM SIMILAR HANDS WITH ADAPTIVE WEIGHTING
We detail our contrastive learning approach (see Fig. 3 learning from mined similar hands with adaptive weighting.
The contrastive learning is designed to align positive samples (I, I+) in the feature space, constructed in Sec. 3.2 and the rest of negative samples are pushed apart. Following (Chen ct al., 2020; |Sprr et al. 2021), ) we treat all mini-batch samples other than the corresponding pos- itive samples as negative samples T- Feature extraction performed by two learnale compo-
Overview:
4

---

Published as a conference paper at ICLR 2025
Projection
feature extraction
Encoder
2
Adaptive weighding
B()
9
T(1*)
weighting
Maximize alignment
Wpos
distonce
Projection head
feature extraction
Encoder
2
T
T(I)
distance
E()
Viter
weighting
Minimize alignment
205
weight
W neg
Adaptive weighting
Projection head
Encoder
2
feature extroction
E(-)
T(T)
Figure 3: Overview our SiMHand. Starting from the left, hand images (1, I-) corresponding 2D keypoints are input to the model. After applying random augmentations through transformation T, both the images and 2D keypoints are spatially transformed. The altered 2D) keypoints then used to compute adaptive weights wpos and Wasg. which guide contrastive learning by strengthening or weakening the alignment between positive and negative samples.
I-) and their
nents: encoder E(-) and projection head g(-), which indicates the entire model as f= oo E. The extraction is combined with image augmentation T, which formulated as z = S(T(T)) and zt f[T((++ )). Applying geometric transformations (e.g., rotation) in can cause misalignment between the image and feature spaces; we correct such an error with the inverse transformation T as (Spurr et al. [021]. After applying the inverse transformation the feature z, we obtain a feature
2 T (z), where geometry is aligned to the original images. As such, all anchor, positive, and negative samples are encoded as 2, 2 +, and 2 respectively.
During learning from our similar hands, propose an adaptive weighting per pair focus more on informative samples that provide greater discriminative information. The assigned weights are computed by the predefined similarity metric in Sec. 3.2 Given pre-processed keypoints for two samples within the mini-batch, k,, k2, the weight w is computed by linear scaling with the Euclidean metric D(.,) as
Adaptive weighting:
dmax D(ki, k2)
W
(2)
dmxx
dman
where dmin dmux are the minimum and maximum distances within the mini-batch. This assigned weight u dynamically changes according to the sample statistics in the mini-batch, enabling adaptive attention per iteration.
To address the distinction between positive and negative sample weighting, we introduce separate weighting terms for positive and negative pairs. Specifically, Wpos corresponds the weight assigned to positive pairs, while Wneg is used for positive-negative pairs.
Contrastive loss with weighting: We finally formulate contrastive learning with the proposed weighting scheme. We assume that mini-batch contains 2N samples in total, including N query samples and their corresponding N positive samples. We introduce separate weighting terms for positives (I, I+) and negatives (I, I-) as Wpes and Whegs respectively. With these weights, strastive learning loss based on the NT-Xent loss (Chen et al. 2020) is formulated as:
exp (Wpss sim (2,, )/7/
C, - -loo
(3)
Σ k= [[ exp (Wneg sim(2. (2, 2 )/7)
5

---

Published as a conference paper at ICLR 2025
Here is temperature parameter, sim(z, 8) adaptive weighting enables considering the importance separately for positive and negative samples, while closer samples are assigned with higher weights and more distant ones receive lower weights.
x²a is the cosine similarity function. Overall, our |
EXPERIMENTS
In this section, we compare our method with existing baselines for pre-training of the 3D hand pose estimation and conduct ablation experiments to support the validity of our approach. We begin by providing detailed explanation of the dataset and experimental setup (Sec. (.1). Next, we demon- strate that our model achieves competitive performance compared with existing methods (Sec. 4.2). Following this, we present the results of ablation studies on weighting design in the pre-training phase (Sec. 4.3). Finally, visualizations are used to illustrate the superiority and efficiency of our approach (Sec. 4.4).
4.1 EXPERIMENTAL SETUP
Pre-training datasets: We curate a large collection of hand images from two major video datasets, Ego4D (Grauman et al. 2022) and 100DOH (Shan al. 2020), featuring egocentric and exocentric views respectively. From Ego4D, a vast egocentric video dataset with 3,670 hours of footage, we extracted 1. .OMM hand images from 8K videos. Similarly, from the exocentric dataset 100DOH, which includes 131 days of YouTube footage, we extract 1.0M hand images from 20K videos. These extensive datasets provide diverse hand-object interactions across different views. We also prepare pre-training data with varying amount. "Exo-X" and "Ego-X" denote 100DOH and Ego4D datasets with X images selected randomly (e.g., X = 50K, 100K, .... 1M, 2M). "Eoo&Exx-2M"" shows our final set combining both datasets with full images (ie., 1.0M for each).
Fine-tuning datasets: We conduct fine-tuning experiments on three datasets with 3D hand pose ground truth in various data size and viewpoints: exocentric datasets from FreiHand (Zimmermann al. 2019) and Dex YCB (Chao et al., 2021). and an egocentric dataset AssemblyHands (Ohkawa et al. 2023b). FreiHand consists of 130.2K training frames and 3.9K test frames, with both green screen and real-world backgrounds. DexYCh contains 325.3K training images and 98.2K test im- ages, focusing natural hand-object interactions. Assembly the largest of the three, in- cludes 704.0K training samples and 109.8K test samples, collected in object assembly scenarios. Following (Spurr et al. (021), we prepare 10% of the labeled FreiHand dataset, which is denoted as "FreiHand"" " especially used for ablation studies. This allow us to assess the performance in a limited supervision setting.
Implementation details:
For similar hands mining, we choose the PCA embedding size as D = 14. For the pre-training framework, we use ResNet-50 (He al.] 2016) the encoder. Throughout the pre-training phase, all models are trained using LARS You et al. 2017) with ADAM (Kingmal & Ba 2014) optimizer, with the learning rate of 3.2c-3. Following (Spurr al. (021). SimCLR employs scale and color jitter as image augmentation, while PeCLR and SiMHand utilize scale, rotation, translation, and color jitter. We use resized images with 128 x 128 as the input. We set the temperature parameter 7 of contrastive learning as 0.5. We use 8 NVIDIA V100 GPUs with a batch size of 8192 for pre-training.
For fine-tuning, we initialize our model with the pre-trained encoder E(-) and then fine-tune with a 3D pose regressor on the labeled datasets. The 3D regressor involves 2D heatmap regression and 3D localization heads, similar to DetNet (Zhou et al. 2020). We use single NVIDIA V100 GPU with a batch size of 128. We provide more additional details in supplement.
We use the following evaluation metrics: the mean per joint position error (MPJPE) in millimeters, which compares model predictions against ground-truth data, and the percentage of correct keypoints based on the area under the curve (PCK-AUC), which measures the proportion of predicted keypoints that fall within a specified distance (20mm to 50mm) from the ground truth with varying thresholds.
Evaluation:

---

Published as a conference paper at ICLR 2025
Table 1: Comparison with the state of the art. We show 3D hand pose estimation accuracy (MPJPEL) on the FreiHand (Exo) Zimmermann et al. 2019). Dex (Exo) (Chao et al. 2021) and AssemblyHands (Ego) (Ohkawa al. 2023b) The best results are highlighted bold, and the second-best results are underlined. SiMHand achieves the best results across various datasets.
FreiHand (Exo)
DexYCB (Exo)
AisemblyHands (Ego)
Method
Pre-training
MPJPE
PCK-AUC1
MPJPE 4
PCK-AUC t
MPJPE
PCK-AUC
w/o pre-training
19.21
85.61
19.36
84.80
19.17
85.61
Exn-IM
19.30
85.36
20.13
83.75
20.01
84.21
SimCLR
Ego-IM IM
19.36
85.09
20.22
83.50
20.32
83.85
2M
20.07
84.32
21.09
82.25
21.24
82.29
Exo-IM
19.58
84.71
18.39
86.33
19.12
85.64
PeCLR
Ego-IM
19.07
85.62
85,40
19.20
E
18.19
86.76
18.06
86.82
18.88
86,03
Exo-IM
16.73
88.66
17.34
87.84
18.50
86.56
SiMHand
Ego-IM
16.15
89.48
16.99
88.34
18.26
86.95
(Ours)
Ego&Exo-2M
15.79
90.04
16.71
88.86
18.23
86.90
Method
Pre-training
FreiHand
SimCLR-IM
4PJPE
PCK-AUCT
PeCLR-1M
w/o pre-training
48.19
49.17
SiMHHndd-1M
SimCLR
53.94
42.54
SiMHand 2M
PeCLR
Ego-5OK
47.42
49.85
SiMHand
35.32
63.35
MPJPE (J))
SimCLR
$3.49
43.12
PeCLR
Ego-100K
46.00
51.50
SiMHand
31.06
68.66
SimCLR
49.91
47.61
PeCLR
Ego-500K
43.18
$4.15
SiMHand
28.27
72.97
SimCLR
46.17
50.62
PeCLR
Bgo-IM
34.42
64.93
80%
SiMHand
23.68
79.62
%6 of labeled training data
Table 2: Comparison with different pre-training data sizes. indicates that use small amount of train- ing data for fine-tuning to validate the effectiveness of the pre-trained model. Our method demonstrates lead- ing advantage across all pre-training data scales.
Figure 4: Comparison with different data availability in fine-tuning on Frei- Hand. Variations in the percentage of labeled data correspond to different sub- sets of the fine-tuning dataset, following the experimental design in (Spurr et al,, 2021)
4.2 MAIN RESULTS
We compare our method with previous works for 3D hand pose estimation (Tab. I To make fair comparison, we evaluate all pre-training datasets the same size against previous methods.
Comparison to contrastive learning methods: We compare our pre- pre-training method with previous methods (Chen et al. 2020; Spurr al. (02T)) 3D hand pose estimation (Tab. We observe that our method significantly outperforms SimCLR and PeCIR across various datasets under the equal pre-training data setups. When we compare our method against a randomly initialized model (w/o pre-training), SiMHand improves performance by 17.7% over the scratch bascline.
In more details, our approach achieves a 15.31% improvement over previous methods PeCLR with Ego- 1M pre-training on the FreiHand. We observe that SimCLR shows limited performance com- pared to the random initialization. This suggests pre-training without geometric prior (i.e., without geometric augmentation) does not always help hand pose estimation, requiring spatial keypoint gression. In contrast, our method demonstrates significant performance gain on larger datasets, with 10.53% gain Dex Dex YCB and 4.90% improvement AssemblyHands compared to PeCLR.

---

Published as a conference paper at ICLR 2025
Table 3: Ablation study of proposed modules. We compare with and without our proposed mod- ules in different methods. The experimental results demonstrate the generality of our method.
Method
Proposals
FreiHand
(Pre-training size)
Similar hands
Adaptive weighting
MPJPE
PCK-AUC 1
SimCLR
53.49
43.12
(Ego-100K)
X
52.58 (1.8% 4)
44.70 (1.58% 1)
PeCLR
X
46.00
51.50
(Ego-100K)
X
44.61 (3.0% })
53.37 (1.87 % 1)
SiMHand
V
31.06
68.66
(Ego-100K)
V
28.84 (7.18 %
71.07 (2.41 %
These results confirm that model consistently achieves superior performance across various fine-tuning datasets.
Furthermore, pre-train all methods the joint pre-training datasets (Ego&EXx-2M), Our ap- proach further improves over the state-of-the-art method (PeCLR), achieving improvements of 13.19%, 7.4%. and 3.4% on the FreiHand, DexYCB, and AssemblyHands, respectively. Compared to the pre-training with IM samples (Ego-IM), doubling the pretraining data with Ego&Exo-20 results in 2.28% improvement on the FreiHand dataset. Notably, our method shows particular strength in effectively handling larger, more varied datasets. This robust performance demonstrates that our approach is highly effective and reliable for hand pose pre-training.
Ego & Exo view analysis: We evaluate how pre-training with egocentric views (Ego4D) and exo- centric views (100DOH) affects the performance in datasets with their corresponding views, namely AssemblyHands for egocentric and FreiHand and Dex for exocentric views. Interestingly, matching pre-training viewpoints does not consistently enhance performance, indicating that the view gaps have limited effects. Instead, factors like dataset diversity and the characteristics of pre- training methods are more crucial in boosting performance. Combining the two datasets (the last row of Tab. T leads to the best performance in all three datasets, underscoring the potential of enriching data diversity with various camera views.
4.3 ABLATION EXPERIMENTS
This section presents ablation studies SiMHand, focusing four aspects: 1) pre-training dataset size, 2) fine-tuning dataset size, 3) adaptive weighting, and 4) Top-K similar hands. First, we exam- ine the size of the pre-training dataset using various methods, showing that approach maintains superior performance across different sizes (Tab.2). Second, inspired by (Zimmermann et al. 2019), explore fine-tuning dataset size, demonstrating significant gains even with limited data (Fig. Furthermore, we also highlight the adaptive weighting design, which consistently outperforms com- parison methods (Tab. 3). Finally, conduct ablation analysis according different levels of similarity in the assigned positive hand pairs. (Tab.
Effect of pre-training data size: We study results with different sizes of pre-training data, namely 50K, 100K, 500K, and IM in Tab. 2 The results demonstrate that SiMHand reliably outperforms the other methods across all settings, with improvement as the pre-training data size increases. With changes in the size of the rre-trainig data from 50K to 1M, SiMHand achieves reduction in MPJPE from 35.32 23.68. The useful insights we can gather from this table include: 1) The SiMHand method holds a leading advantage across various pre-training size. 2) As the size of the pre-training dataset increases, the improvement for fine-tuning with limited labels substantial.
Effect of fine-tuning data size: Fig. 4 iillustrates the experiment under different proportions of la- beled fine-tuning data, namely 10%, 20%, 40%, and 80% FreiHand. Note that we denote methods with "-1M/2M" as those pre-trained on the Ego-IM and the EgooExo-2M sets, respectively. The results show that SiMHand- IM brings error reduction, achieving remarkably lower MPJPE scores with merely 10% of labeled data. SiMHand- 1M delivers the best performance of fine-tuning data, compared to SimCLR- - and PeCLR- M.. SiMHand 2M further shows im- provement over SiMHand-INM while the gains become marginal labeled data increase. From
different size
8

---

Published as a conference paper at ICLR 2025
Table 4: Pre-training performance at different similarity ranks (Top-K). It can be seen that as the similarity rank increases, the pre-training performance deteriorates.
Method
FreiHand
Top-K
(Pre-training size)
MPJPE +
PCK-AUCT
Top-1
31.06
68.66
Top-2
31,46
67.89
Top-5
31.85
67.20
Top-10
31.87
67.18
SiMHand
Top-50
31,53
67.59
(Ego-100K)
Top-100
31.54
67.70
Top-500
32.61
66.76
Top-1000
34.05
65.14
Top-5000
35.34
62.79
this analysis, can draw two key conclusions: 1) The improvement resulting from an increase of pre-training data becomes less significant as the amount of fine-tuning data increases; 2) SiM- Hand maintains strong advantage in scenarios with limited labeled data, particularly when larger pre-training data are used.
Effeet of adaptive weighting: We validate the proposed adaptive weighting and generality when applied to the other methods in Tab. 3 On the Ego-100K pre-training set, the MPJPE scores af- ter adaptive weighting decrease by 1.8% and 3.0% for SimCLR and PeCLR, respectively, while 1.58% and 1.87%. This indicates that the proposed weighting excels in its applicability to various pre-training methods. our SiMHand method, applying adaptive weighting reduces MPJPE from 31.06 28.84, 7.18% decrease, while PCK-AUC improves from 68.66 to 71.07, a 2.41% increase. We find the effectiveness of the proposed weighting when combined with the mined similar hands.
PCK-AUC increases
We test pre-training with different similarity levels of posi- tive samples in Tab. [ As illustrated in Fig. 2 we can sample similar pairs according the distance 5000), where Top-I is used to produce our final results. The performance trend is initially subtle and somewhat fluctuating (Top-I~100) but becomes increasingly pronounced after T00- This indicates that as the similarity between positive samples increases, the global trend decreases accordingly. Notably, using Top-5000 similar hand samples as positive samples de- creases the MPJPE by 13.78% compared to Top-1. This study provides two insights: 1) Similar samples with subtle noisiness (e.g., 1~100) exhibit minimal variation in performance, indicating that slight differences in similarity within this range do not significantly impact the pre-training outcome. This suggests that the model is robust to minor variations when the positive samples are highly similar. 2) The results support the validity of using Top-1 positive samples to produce final results, as they consistently exhibit the best performance. This highlights the importance of selecting the most similar samples in contrastive learning.
Learning from Top-K similar hands:
ranking (e.g., K 1,
4.4 VISUALIZATION
In this section, we compare the fine-tuning results of various pre-training methods through detailed visualizations on different datasets (Fig. 5). The pre-training model trained on the Ego&Exo-2M dataset and tunet on the FreiHands (Zimmermann et al., (2019) and Dex YCB (Chao et al. 2021) datasets, respectively. We provide additional visualization in the supplementary material.
From the left four columns of Fig. 5 the visualization results show that SiMHand performs better pose estimation, with results closer to the ground truth, compared to the other methods in Frei- Hands (Zimmermann et al., 22119) dataset. In particular, SiMHand outperforms the other methods in challenging environments, such as those with varying lighting conditions, by better capturing hand poses. These visual outputs highlight its robustness across various scenarios, solidifying its potential for real-world applications.
As shown in the right four columns of Fig. of Dex YCB (Chao et al.) (02I) dataset using red circles. The results show that SiMHand is more effective in tackling occlusion problems. Our pre-training method effectively addresses partially
highlight the occluded regions in the original images

---

Published as a conference paper at ICLR 2025
FreiHand
DexYCB
Original
GT
SiMHand
PeCLR
SimCLR
w/o PT
Figure Visualization of FreiHand ZZimmermann al. 2019 and DexYCB (Chao et al,, 2021). The first four columns the left display the results for FreiHand, while the last four columns on the right show the results for Dex YCB (GT: Ground Truth; PT: Pre-training). It can observed that SiMHand pre-training method achieves better results.
occluded images by utilizing similar, though not identical, hand images, where the occluded parts in the query image may be visible in the corresponding similar hand image, and vice versa.
CONCLUSION
We introduce SiMHand, contrastive learning framework for pre-training 3D hand pose estimators by mining similar hand pairs from large-scale in-the-wild images. Our approach leverages similar hand pairs from diverse videos, significantly enhaneing the information gained during pre-training compared with existing methods. Experiments show that pre-training method achieves com- petitive performance 3D hand pose estimation across multiple datasets, outperforming previous pre-training approaches and demonstrating the benefits of large-scale pre-training with in-the-wild images. We hope this work can lay a foundation for future research on pre-training of 3D hand pose estimation.
ACKNOWLEDGMENTS
This work was supported by the JST ACT-X Grant Number JPMIAX2007.111 ASPIRE Grant Num- ber JPMJAP2303, JSPS KAKENHI Grant Number JP24K02956, JP22KF0119 19, and NSFC Grant Number 62376090.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
R. Arandjelovic, Gronat, A. Torii, Pajdla, and Sivie. Netvlad: CNN architecture for weakly supervised place recognition. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5297-5307, 2016.
G. Berton, Masone. and Caputo. Rethinking visual geo-localization for large-scale applications. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4878- 4888, 2022.
Cai, I.. Ge, J. Cai, J. Yuan. Weakly-supervised 3D hand pose estimation from monocular RGB images. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 678-694, 2018.
M. Caron, I. Misra, Mairal, Goyal, Bojanowski, and Joulin. Unsupervised learning of visual fea- tures by contrasting cluster assignments. In Proceedings of the Advances Neural Information Processing Systems (NeurlPS), pp. 9912-9999999 222-9924,222 2020.
M. Caron, H. Touvron, Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties self- supervised vision transformers. In Proceedings of the IEEE/CVE International Conference on Computer Vision (ICCV), pp. 9650-9660, 2021.
Y.W. Chao, W. Yang, Y. Xiang, P. Molchanov, A. Handa, J. Tremblay, Y. S. Narang, K. Van Wyk, U. Iqbal, S. Birchfield, J. Kautz, and D. Fox. YexYCB: A benchmark for capturing hand grasping of objects. In Proceedings the the IEEF/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 9044- 9053, 2021.
H. Chen, B. Lagadec, and F. Bremond. Ice: Inter-instance contrastive encoding for unsupervised person re- identification. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 14960-14969, 2021a.
H. Chen, Y. Wang, B. Lagadec, A. Dantcheva, and F. Bremond. Joint generative and contrastive learning for unsupervised person re-identification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2004-2013, 2021b.
Chen, Kornblith, M. Norouzi, and G. E. Hinton. A simple framework for contrastive learning of visual representations. In Proceedings of the International Conference Machine Learning (ICML), volume 119, pp. 1597-1607, 2020.
X. Chen and K. He. Exploring simple siamese representation learning. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15750-15758, 2021.
Y. Chen, Z. Tu, D. Kang. L. Bao, Y. Zhang. X. Zhe, R. Chen, and J. Yuan. Model-based 3d hand reconstruction supersupe learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10451-10460, 2021c.
Chopra, R. Hadsell, and Y. LeCun. Learning similarity metric discriminatively, with application face verification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 539-546, 2005.
Fan, Ohkawa. L. Yang, Lin. Zhou, S. Zhou, Liang, Z. Gao, Zhang. X. Zhang, F. L. Zheng, F. Lu, K. A. Zeid, B. Leibe, J. On, S. Back, A. Prakash, S. Gupta, K. Hc, Y. Sato, O. Hilliges, H. J. Chang, and A. Yao. Benchmarks and challenges in pose estimation for egocentric hand interactions with objects. In Proceedings of the European Conference Computer Vision (ECCV), 2024.
L. Ge, 7. Ren, Y. Li, 7. Xue, Y. Wang, J. Cai, and J. Yuan. 3D hand shape and pose estimation from a single RGB image. In Proceedings of the IEEE/CVF Conference Computer Vision and Patiern Recognition (CVPR), pp.
2019.
Y. Ge, H. Wang, F. Zhu, R. Zhao, and H. Li. Self-supervising fine- dieeggrined region similarities for large-scale image localization. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 369-386, 2020.
K. Grauman, A. Westbury, E. Byrne Z. Chavis, A. Furnari, R. Girdhar, J. Hamburger, H. Jiang, M. Liu, X. Liu, M. Martin, T. Nagarajan, I. Radosavovic, S. K. Ramakrishnan, F. Ryan, J. Sharma, M. Wray, M.g Xu, E. Zhongcong Xu, C. Zhao. Bansal, D. Batra,
Cartillier, S. Crane, T. Do, M. Doulaty, A. Era- palli, C. Feichtenhofeer A. Fragomeni, Q Fu, C. Fuegen, A. Gebreselasie, C. Gonzalez, J. Hillis, X. Huang, Y. Huang, W. Jia, W. Khoo, J. Kolar. S. Kottur, A. Kumar, F. Landini, C. Li, Y. Li. Z. Li, K. Mangalam, R. Modhugu, J. Munro, T. Murrell, T. Nishiyasu, W. Price, P. R. Puentes, M. Ramazanova, I. Sari, K. So- masundaram, A. Southerland, Y. Sugano, R. Tao, M. Vo, Y. Wang, X. Wu, T. Yagi, Y. Zhu, P. Arhelaez,
11

---

Published as a conference paper at ICLR 2025
D. Crandall, D. Damen, G. M. Farinella. B. Ghanem, V. K. Ithapu, C. V. Jawahar, H. Joo. K. Kitani, H. Li, R. Newcombe, Oliva, H. Soo Park, M. Rehg. Sato, J. Shi, M. Z. Shou, A. Torralba, Lo Torresani, M.i Yan, and J. Malik. Ego4D: Around the world in 3, 000 hours of egocentric video. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 18973 11999,, 2022.
J. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar, B. Piot, K. Kavukcuogiu, R. Munos, and M. Valko. Bootstrap your own latent: A new approach self-supervised learning. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), pp. 21271-21288 2020.
S. Han, P.-C. Wu, Y. Zhang, B. Liu, L. Zhang. Z. Wang. WiSi, P. Zhang, Cai, T. Hodan, R. Cabezas, L. Tran, M. Akbay, T.HH. Yu, Keskin, and Wang. UmeTrack: Unified multi-view end-to-end hand tracking for VR. In In Proceedings of the ACM SIGGRAPH Asia Conference, pp. 50:1-50:9, 2022.
Hausler, S. Garg, M. Xu, M. Milford, and T. Fischer. Patch-netvlad: Multi-scale fusion of locally-global descriptors for place recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14141-14152, 2021.
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognation (CVPR). pp. 9729 9738, 2020.
Y. Huang, L. Yang, and Y. Sato. Weakly supervised temporal sentence grounding with uncertainty-guidee self-training. In Proceedings of the IEÉE/CVF conference on computer vision and pattern recognition, pp. 18908-18918, 2023.
X. Jie, Chen, Y. Ren, X. Shi, H. Shen, G. Niu, and X. Zhu, Self-weighted contrastive learning among multiple views for mitigating representation degeneration. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), pp. 1119-1131, 2024.
D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412661412,6990, 2014.
R. Liu, T. Ohkawa, M. Zhang, and Y. Sato. Single-o--dull-iies adaptation for egacentric 3d hand pose esti- mation. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 677-686, 2024.
S. Liu, H. Jiang. J. Xu, S. Liu, and X. Wang. Semi-supervised 3D hand-object poses estimation with interactions in time. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14687-14697, 2021.
Lugaresi, J. Tang, H. Nash, C. McClanahan, E. Uboweja, M. Hays, and F, Zhang al. Mediapipe: framework for building perception pipelines. arXiv preprint
2019.
G. Moon, S.-L. Yu, H. Wen, T. Shiratori, and K. M. Lee. InterHand2 6M: A dataset baseline for 3D interacting hand pose estimation from a single RGB image. In Proceedings of the European Conference on Computer Vision (ECCV), pp.
2020.
T. Ohkawa, Y.J. Li, Q. Fu, R. Furuta, K. M. Kitani, and Y. Sato. Domain adaptive hand keypoint and pixel localization in the wild. In Proceedings of the Luropean Conference Computer Vision (ECCV), pp. 68-87, 2022.
T. Ohkawa, R. Furuta, and Y. Sato. Efficient annotation and learning for 3D hand pose estimation: A survey. International Journal on Computer Vision (UCV), 131:3193-3200 2023a.
Ohkawa, K. He, F. Sener, T. Hodan, L. Tran, and C. Keskin. AssemblyHands: Towards egocentric activity understanding via hand pose estimation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 12999- 13008, 2023b.
J. Park, Y. Oh, G. Moon, H. Choi, and K. M. Lec. Handocenet: Occlusion- 3d hand mesh estimation network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1486-1495. 2022.
Y. Qin, Y.-. Wu, S. Liu, H. Jiang, R. Yang. Y. Fu, and X. Wang. DexMV: Imitation learning for dexterous manipulation from human videos. In Proceddnggs of the European Conference on Computer Vision (ECCV), volume 13699, pp. 570-587, 2022.
12

---

Published as a conference paper at ICLR 2025
A. Radford, J. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin. J. Clark, and G. Krueger. Learning transferable visual models from natural language supervision. Proceedings of the International Conference on Machine Learning (ICML), pp. 8748-8788 8763, 2021.
F. Schroff, D. Kalenichenko, and J. Philbin. Facenet: A unitied embedding for face recognition and clustering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 115-823, 2015.
F. Sener, D. Chatterjee, D. Shelepov, K. He, D. Singhania. R. Wang. and A. Yao. Assembly101: A large- scale multi-view video datasct for understanding procedural activities. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 21096-21 2116,, 2022.
D. Shan, J. Geng. M. Shu, and D. Fouhey. Understanding human hands in contact at internet scale. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9866-9875. 2020.
K. Sohn. Improved deep metric learning with multi-class n-pair loss objective. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), pp. 1857-1865, 2016.
H. Song, Y. Xiang. Jegelka, and S. Savarese. Deep metric learning via lifted structured feature embedding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 4004 4012, 2016.
A. Spurr, U. Iqbal, P. Molchanov, O. Hilliges, and J. Kautz. Weakly supervised 3D hand pose estimation via biomechanical constraints. In Proceedings the European Conference on Computer Vision (ECCV), pp- 211 121-228,220 2020.
A. Spurr, A. Dahiya, X. Wang. Zhang, and O. Hilliges. Sele-vupe 3d hand pose estimation from monocular rgb via contrastive learning. In Proceedings of the IEEE/CVF International Conference on Com- puter Vision (ICCV), pp. 11230-1 1239,, 2021.
K. Tango, T. Ohkawa, R. Furuta, and Y. Sato. Background mixup data augmentation for hand and object-in- contact detection. In Proceedings the European Conference Computer Vision Workshops (ECCVW), 2022.
Tse, K. Kim, A. Leonardis, and H. Chang. Collaborative learning for hand and object reconstruction with attention-guic guided graph convolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1664-1674. 2022.
Y. Wen, H. Pan, T. Ohkawa, L. Yang, J. Pan, Y. Sato, T. Komura, and W. Wang. Generative hierarchical temporal transformer for hand action recognition and motion prediction. arXiv preprint 17X.v.2311.17366, 2023.
M.-Y. Wu, P.-W. Ting, Y.-H. Tang, E. Chou, and L.-C.. Fu. Hand pose estimation in object-interaction based on deep learning for virtual reality applications. Journal of Visual Communication and Image Representa- tion, 70: 1020022 04 2020.
F. Xiong, B. Zhang, Y. Xiao, Z. Cao, T. Yu, J. T. Zhou, and J. Yuan, A2I: anchor-to-joint regression network for 3D articulated pose estimation from single depth image. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 793 802, 2019.
Yang, S. Chen, and A. Yao. Semihand: Semi-supervised hand pose estimation with consistency. In Proceed- ings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 1 11364-11373,200 2021.
Y. You, I. Gitman, and B. Ginsburg. Large batch training of convolutional networks. arXiv preprint arXiv: 73888888, 2017.
H. Zhang, Y. Hou, W. Zhang, and W. Li. Contrastive positive mining for unsupervised 3d action representation learning. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 36-51, 2022.
X. Zheng, C. Wen, Xue, P. Ren, and J. Wang. Hamuco: Hand pose estimation via multiview collaborative superviedd learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 20763-20773, 2023.
Y. Zhou, M. Habermann, W. Xu, I. Habibie, C. Theobalt, and F. Xu. Monocular real-time hand shape and motion capture using multi-modal data. In Proceddiggs of the IEFF/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 5346-5355, 2020.
13

---

Published as a conference paper at ICLR 2025
A. Ziani, Z. Fan, M. Kocabas, S. J. Christen, and O. Hilliges. Tempelr: Reconstructing hands via tome-coherent contrastive learning. In Proceedings of the International Conference 3D Vision (3DV), pp. 627-636, 2022.
Zimmermann, D. Ceylan, Yang. B. Russell, J. Argus, and T. Brox. FreiHAND: dataset markerless capture of hand pose and shape from single RGB images. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 813-822, 2019.
14

---

Published as a conference paper at ICLR 2025
Step1: Preprocessing:
Step2: Hand crops collection:
Step3: Similarity Calculation & Ranking:
Pase estimator
Hand datactor
Anchor
Hand bbox
Ego4D Datasets [SK]
PCA
(tos=1)
0.9
Hand detector
Hand bbox
100DOH Datasets [23K]
(16e-1) 1)
Pool of hand images in the wild
Top-1
Top-2
Top-K
Figure 6: Overview of data preprocessing and similar hands mining. This image illustrates a three-step process for SiMHand pre-training using datasets from Ego4D and 100DOH. Step involves preprocessing the datasets to extract relevant frames. Step 2 employs hand detector to crop hand regions from these frames, creating a diverse pool of hand images in the wild. Step calculates similarity and ranks the images using pose estimator and PCA, producing a sorted list. of hand poses, from the most similar to the least similar to a given anchor pose.
APPENDIX
6.1 CONSTRUCTION OF LARGE-SCALE --THE--ILL HAND DATABASE
This section presents our method for constructing a large-scale hand image dataset by extracting and processing hand images from various video datasets. We outline key preprocessing steps, including 1) preprocessing, 2) hand region detection, and 3) similarity calculation & ranking.
Preprocessing: We prepare two large-scale video datasets: Ego4D, containing 8k frames, and 100DOH, with 23k frames, both sampled fps. As shown in Fig. first-person and third-person hand images exhibit significant differences.
Hand region detection: After extracting frames from Ego4D and 100DOH, we use a lightweight, fixed-weight network to detect hand regions via bounding boxes. Specifically, we adopt the method from (Shan et al. 2020) and store detected bounding boxes in sequence. This step constructs a large-scale hand image dataset as Tango et al. (2022).
Similarity calculation & ranking: Once the hand image dataset is built, we use a lightweight, fixed-weight network to extract raw keypoints for each sample via MediaPipe (Lugaresi et al. 2019). To reduce noise, we apply PCA as described Sec. 3.1 We then compute similarity scores for given query image / using Eq. and rank the remaining samples accordingly. This process yields large-scale set of in-the-we wild hand images with similar characteristics. For instance, in Ego4D, given query sample I, retrieve all similar hand images and construct ranked sequence, referred to as "Top-K". The Top-1 image in this sequence serves as the positive sample J for contrastive learning, enhancing the effectiveness of SiMHand pre-training. As shown in Tab. 4 our experiments validate that selecting Top-1 as the positive sample I+ is the optimal strategy.
6.2 FINETUNE FOR 3D HAND POSE ESTIMATION
In the fine-tuning stage, we discard the projection head and fine-tuning only the encoders. We load the pre-training model weights into a heatmap-based 31) hand pose estimationand predition method: DetNet(Zhou et al., 2020). To train DetNet, we utilize comprehensive loss function designed to optimize both 2D pose estimation and 3D spatial localization. The loss function is defined as:
Cheat + Lioe + Laelta + Creg
(4)
where Lheat ensures that the predicted heatmaps H align closely with the ground truth heatmaps HT Lloe and Ldelta measure the discrepancies between the predicted location maps L and delta maps and their corresponding ground truth GT and DCT with HTT weighting these discrep- ancies to focus the maxima of the heatmaps. Additionally, frog is L.2 regularization term to
15

---

Published as a conference paper at ICLR 2025
FreiHand
Method
Pre-training size
MPJPR +
PCK-AUC
PeCLR
47.42
49.85
TempCLR
Ego-50K
45.17
52.40
SiMHand
35.32
63.35
PeCLR
46.00
51.50
TempCLR
Ego- Ego-100K 100K
44.54
53.28
SiMHand
31.06
68.66
Table 5: Comparison with the TempCLR method." **' indicates that we use a small amount of training data for fine-tuning to validate the effectiveness of the pre-trained model. TempCLR out- performs PeCLR by a modest margin, whereas SiMHand achieves significant performance im- provement over TempCLR.
prevent overfitting. Note that after passing through the encoder, we made simple adjustments to the model, applying some upsampling to the features to fit the input.
This multi-task learning framework enables the network to simultancously learn pose features from 2D images and spatial information from 3D data, enhancing the accuracy and robustness of detection in real-world applications. For more details on fine-tuning, please refer to the (Zhou et al., 2020).
6.3 COMPARISON WITH TEMPCLR METHOD
We conduct experimental comparison with the TempCLR (Ziani et al. 2022)
method. Temp-
CLR proposes pre-training framework for 3D hand reconstruction using coherent contrastive learning and demonstrates better performance compared to PeCLR (Spurr et al.) (02I). Although TempCLR primarily focuses on reconstruction tasks, the parametric model it uses can also output 3D pose results, making it valuable to further compare method with TempCLR.
However, TempCLR has certain limitations in data collection and the effectiveness of contrastive learning. First, TempCLR treats hands from adjacent frames positive samples during training. In dynamic egocentric videos, hand occlusions or detection failures often lead to missed hand crops in neighboring frames. In addition, images from adjacent frames typically lack background diversity, limiting the contribution of positive sample pairs formed from neighboring frames in contrastive learning.
In contrast to TempCLR, our method, SiMHand, significantly improves performance. SiMHand leverages similar hand images, which provide richer diversity in features, including various types of hand-object interactions, diverse backgrounds, and varying appearances. These features allow SiMHand to effectively increase the diversity of positive samples contrastive learning, resulting in superior pre-training performance.
We further validate our approach on two different size of pre- gre-training data, consisting of 50K and 100K hand images from the Ego4D dataset (Grauman et al. (022). Tab. 5) shows the significant progress made by SiMHand compared to TempCLR and PeCLR.
From the experimental results, TempCL. demonstrates better performance than PeCLR, which matches the conclusion of the original paper. However, SiMHand provides more valuable posi- tive samples for contrastive learning, leading to better results during the fine-tuning phase of 3D hand pose estimation tasks.
6.4 COMPARISON WITH WEAKLY-SUPERVISED LEARNING SETTING
We compare our method with a weakly-supervise learning setting that uses 2D noisy keypoints assigned on -n-the-wild images. In weakly-supervised learning setting, noisy 2D keypoints are directly used as supervision signals during network training. The model treats these 2D keypoints targets, computing the loss between the predicted keypoints and the provided 2D keypoints (e.g., heatmap-based loss). However, our experiments reveal that directly conducting joint training on labeled and unlabeled data results in degraded performance due the noise and unreliability of
16

---

Published as a conference paper at ICLR 2025
FreiHand
Setting
Unlabeled data
MPJPE +
PCK-AUC 1
Weakly-supervised
Bg0-100K
61.65
33,92
Pre-training & Fine-tuning
Ego-100K 100K
31.06
68.66
Table Comparison with weakly-supervisee learning setting, We observe that directly incorpo- rating noisy labels into the joint training in the weakly-supervised setting leads to decline in model performance, indicating that applying noisy labels for training presents certain challenges.
DexYCB
Method
Backbone
MPJPE +
Xiong et
2019)
ResNe150
25.57
Spurr al.
2020
ResNet50
22.71
Spurr al.
2020
HRNet32
22.26
Tse al.
2022]
ResNet18
21.22
Zhou et al.
(2020)
ResNe150
19.36
SiMIland
ResNet50
16.71
Table 7: Comparison of 3D hand pose estimation methods DexYCB (Chao et al. 2021
GisemblyHandh
Method
Backbone
MPJPE +
Han et al.
2022
ResNet50
32.91
Ohkawa et al.
2023b
ResNet50
21.92
Zhou et al.
2020
ResNet50
19.17
SiMHand
ResNet50
18.23
Table 8: Comparison of 3D hand pose estimation methods on AssemblyHands (Ohkawa et al. 2023b).
the 2D keypoints. As shown in Tab. 6 without any keypoint filtering correction, the weakly- supervised method performs significantly worse than our pre- pre-training setting.
These findings demonstrate that directly incorporating noisy 2D annotations during weakly- supervised training negatively impacts model performance, particularly when the labels contain high levels of noise.
Before designing our pre-training approach, we identified several limitations of the weakly- supervised setting for large-scale, in-the-wild hand data based on prior experience: (1) Data seale constraints: When the amount of noisy hand data is significantly smaller than the noise-free hand training dataset, may provide some improvement but it is hard to guarantee that such noisy la- bels less in larger datasets (e.g., the two million in-the-wild hand images in this study) and (2) Training efficiency issues: Introducing large-scale noisy data significantly prolongs training time and slows convergence. In contrast, pre-training method benefits from such large unlabeled hand images with certain noisiness. This highlights our superiority exploiting pre-training over the weakly-supervisee setting.
6.5 COMPARISON WITH THE OTHER 3D HAND POSE ESTIMATION METHODS
To better assess the value of this work and its position within the broader context, we have included comparisons with other related works in the field of 3D hand pose estimation in this section.
As shown in Tab. and g the comparative results on the YCB (Chao et al. 2021 and As- semblyHands OOkawa al. 2023b) datasets further validate the superiority of our approach across multiple standard datasets, demonstrating the effectiveness our pretraining strategy and its broad potential for real-world applications.
17

---

Published as a conference paper at ICLR 2025
AssemblyHands
GT Original
SiMHand
PeCLR
SimCLR
w/o PT
Figure 7: Visualization of Hand Pose Estimation Results AssemblyHands. Assembly Hands Ohkawa et al. (2023b) is a hand pose dataset captured from a first-person perspective during toy assembly. It can be observed that SiMHand pre-training method achieves better results (GT: Ground Truth; PT: Pre-training).
6.6 VISUALIZATION OF HAND POSE ESTIMATION RESULTS ON ASSEMBLYHANDS
We show the visualization results of hand pose estimation on another dataset, Assembly Ohkawa et al. 2003b). We highlight instances of hand-object occlusion in the data using red circles. As observed with Dex (Chao et al.) (021), SiMHand pre-trained model demonstrates superior performance in handling occlusion during the fine-tuning stage compared the other pre-training methods, showcasing stronger robustness.
6.7 VISUALIZATION OF SIMILAR HANDS
We present the visualization of Top-K similar hand images used create positive pairs. shown in Fig. 8 we visualize a set of Top-K similar hand images. The figure displays the query image alongside its corresponding similar hand sequence (Top-K), K). At the top of Fig. 8, timeline indicates that the images are deliberately sampled from consecutive frames of the same video.
From these visualizations, we derive three key insights: 1) Using adjacent frames from the same video as positive samples in pre-training lacks diversity, as substantial variations may still exist between samples. 2) As the ranking increases, the similarity between hand images decreases sig- nificantly, leading greater differences that may result in inaccurate feature representations during pre-training. 3) Therefore, selecting the Top-1 image is a proper design to assign diverse yet similar positive samples for the query images.
18

---

Published as a conference paper at ICLR 2025
Query image
Top-1
Top-2
Top-5
Top-10
Top-50
Top-100
Top-500
Top- 1000
Top-5000
Figure 8: Visualization of similar hand samples in Top-K. As the ranking increases, the differ- ences between hand samples become more pronounced.
19


Reviews:
[{"review_id": "9HXgKGZeMS", "reviewer": "Reviewer_vbov", "summary": "This paper presents a contrastive learning method for the pre-training of 3D hand pose estimation based on large-scale in-the-wild hand images. A parameter-free adaptive weighting mechanism is introduced in the contrastive learning loss, which not only learns from similar samples but also adaptively weights the contrastive learning loss based on inter-sample distance. Experiments show improved performance compared with existing pre-training methods.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is well written and easy to follow.\n- The motivation of finding similar hands derived from different video domains is technically sound, which can further benefit contrastive learning process from discriminating foreground hands in varying backgrounds.\n- The experimental results in Table 3 demonstrate the generality of the proposed contrastive learning with adaptive weighting mechanism.", "weaknesses": "- TempCLR [1] proposes a pre-train framework for 3D hand reconstruction with time-coherent contrastive learning, and shows better performance compared with PeCLR. Although TempCLR focuses on reconstruction tasks, the used parametric model can output 3D pose results. Therefore, more comparisons with TempCLR would be helpful.\n\n- In the second column of Figure 6, HandCLR demonstrates advanced performance in hand-object occlusion. Does the proposed method exhibit robustness in similar severe occlusion scenarios involving hand-object interactions? More qualitative analysis in datasets like DexYCB or in-the-wild scenarios would be helpful.\n\n- The proposed adaptive weighting mechanism is a straightforward approach that has proven effective; however, it lacks a clear articulation of its motivation, particularly regarding the challenges faced in 3D hand pose estimation tasks as mentioned in the introduction.\n\n[1] Tempclr: Reconstructing hands via time-coherent contrastive learning. In  3DV, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YEW720yTm9", "reviewer": "Reviewer_4Kod", "summary": "- The paper explores pretraining for hand pose estimation using a large number of 2D image samples.\n- It introduces HandCLR, a contrastive learning-based method. This method expands the definition of positive and negative samples by using pairs with similar actions from different sources, improving upon previous methods that relied solely on data augmentation.\n- The authors collected extensive pretraining datasets from 100DOH and Ego4D and studied effective methods for mining similar hand samples.\n- They designed a Top-K sampling strategy for positive and negative samples and implemented adaptive weighting.\n- Experiments show that the proposed method outperforms baselines in both pretraining and downstream finetuning tasks.", "questions": "- Why are the baseline metrics relatively poor? For example, Freihand dataset shows 18+ MPJPE, while recent works (i.e. MobRecon) often achieve <6 PA-MPJPE. Could you explain if Procrustes analysis accounts for such a large performance difference? If the author could explicitly address this performance gap or more clearly explain the difference between the baseline metrics and those of existing fully supervised methods, it would be better.\n- Are the positive sample augmentations identical to those used for query images?\n- Is Figure 4 showing results from the FreiHand dataset?\n- Regarding minibatch construction, the authors mention using 2N samples (N query images and their corresponding positive samples). Using the top-1 method for defining positive samples, could there be cases where a negative sample $I_n$ for query image $I_m$ is actually very similar but not top-1 (e.g., top-K where K>1)? Do the authors have more detailed descriptions of how to increase the discrimination in positive/negative sample sampling, or is it solely addressed through adaptive weighting?\n- How is diversity ensured in the reverse lookup of top-1 samples for each query image? Could there be cases where samples from videos j and k are mutually top-1 similar samples, potentially reducing training diversity by constantly pairing samples from the same two videos?\n- What specific models were used as baselines in Tab.1?\n\n- Since the baselines compared by the author all have open-source code, in order to enhance the reproducibility of the article and the usability for downstream tasks, it is hoped that the author will adhere to what is mentioned in the article and actually release the code", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written and easy to follow.\n- The motivation behind the proposed method is sound, with comprehensive details from data preparation to training.\n- The design of contrastive loss with weighting provides better gradient guidance for samples with different sources and similarities, which is both reasonable and effective.\n- The numerous experiments reflect significant effort by the authors.\n- The experimental section is logical and thorough, demonstrating performance improvements across different datasets and analyzing the impact of training samples, finetuning sample size, and various design modules.", "weaknesses": "- Some presentation issues need improvement\n    - Figure 6 should be updated to remove inappropriate \"bbox\" spelling marks. Additionally, all images in the paper should be replaced with vector versions to prevent blurry text, as seen in Figure 3.\n- The article lacks references and discussions on self-supervised methods. The recent two works, S2Hand and HaMuCo, although not pre-training methods, also attempt to use unlabeled images and 2D off-the-shelf detectors to train 3D hand pose estimation models.\n- Otherwise, the paper is relatively complete with no major weaknesses \n\n[1] Model-based 3D Hand Reconstruction via Self-Supervised Learning\n\n[2] HaMuCo: Hand Pose Estimation via Multiview Collaborative Self-Supervised Learning", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Mo973AFydk", "reviewer": "Reviewer_PnQd", "summary": "This paper addresses the task of self-supervised learning for 3D hand pose estimation from monocular RGB. The authors build on prior work in the area and improve upon it in three main areas: 1) Use of noisy 2D supervision to mine positive samples 2) Adaptive weighting that weighs positive and negative samples based off of their distance of the 2D keypoints 3) Processing and use of Ego4D and 100DOH for self-supervision.\nTheir proposed method first constructs a pose embedding based off of the noisy acquired 2D keypoints using an off-the-shelf predictor. This pose embedding is then used to mine positive samples given a query image.\nThese positive samples are used within the contrastive loss as positive samples, whereas the remaining image in the batch is marked as negative. The positive and negative samples are weighted additionally using weights that are computed based off of the scaled euclidean distance of their 2D keypoints.\nThe self-supervised model is trained on Ego4D and/or 100DOH. Those datasets have been processed using an off-the-shelf hand detector model. Supervised training was done on a variety of supervised datasets. Experimental results show large improvement across all benchmark datasets compared to prior self-supervised models.", "questions": "- How does this work compare to a weakly-supervised approach with noisy annotations?\n- How does this work perform compared to other related work on the tested datasets?\n- Instead of using weights, could one instead not use a more appropriate loss that will automatically lead to larger effects depending on the samples weights? For example, MSE will automatically weight the contributions of more distant samples stronger.\n- L143-144: Why balance the number of left and right hand if they all end up being converted to right-handed images?\n- Eq 1: Why not use the cosine similarity which is more popular for distances in feature space?\n- Fig3: The colored boxes at the end of the model pipeline seems to be in the wrong order. E.g the figure shows positive samples minimizing alignment.\n-L238-239: rough -> noisy\n- Table 1: What is \"baseline\"? This needs to be explained in the image caption\n- Table 1: Why are the worst result of SimCLR in bold? Shouldn't the most performant number be in bold?\n- Not all figures and tables are referred to in text.\n- Table 3: inconsistent capitalization of simclr etc. This also occurs occasionally in the text.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The paper empirically verifies the improvement over prior self-supervised models. Self-supervision is a rather underexplored area in hand pose estimation and can lead to potentially great benefit as foundation models.\n- The improvements are substantial\n- The paper is easy to understand", "weaknesses": "- The method shows great improvement over prior self-supervised methods through the use of noisy 2D annotations. However, its use is a rather involved process: it first needs to be embedded, then used during pre-training before then performing supervised fine-tuning. Instead, why not just use the noisy 2D annotations directly as a form of weak-supervision? In fact, this has been done in prior work [1] and has lead to substantial improvements. In order to properly verify the usefulness of the authors proposed method, there first needs to be a baseline showcasing that the straightforward addition of the noisy 2D annotation during pre-training or supervised training performs comparatively worse. Otherwise why should one employ the authors proposed method? Due to my own experiences in the field, I fear that the weak-supervision approach will outperform the authors proposed approach.\n- The paper does not compare to other related work in the field for which test results on FreiHand, DexYCB and AssemblyHands are available. Without these, we cannot assess properly the value of this work and how it fits in overall.\n\n\n[1] Weakly-Supervised Mesh-Convolutional Hand Reconstruction in the Wild, Kulon et al., CVPR'20", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)