A Dataset of Relighted 3D Interacting Hands
Authors: Gyeongsik Moon
Shunsuke Saito
Weipeng Xu
Rohan Joshi
Julia Buffalini
Harley Bellan
Nicholas Matthew Rosen
Jesse Richardson
Mallorie Mize
Philippe De Bree
Tomas Simon
Bo Peng
Shubham Garg
Kevyn Alex Anthony McPhail
Takaaki Shiratori
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: dataset
3D interacting hands
relighting
TL;DR: We introduce a new dataset of 3D interacting hands.

Abstract: The two-hand interaction is one of the most challenging signals to analyze due to the self-similarity, complicated articulations, and occlusions of hands. Although several datasets have been proposed for the two-hand interaction analysis, all of them do not achieve 1) diverse and realistic image appearances and 2) diverse and large-scale groundtruth (GT) 3D poses at the same time. In this work, we propose Re:InterHand, a dataset of relighted 3D interacting hands that achieve the two goals. To this end, we employ a state-of-the-art hand relighting network with our accurately tracked two-hand 3D poses. We compare our Re:InterHand with existing 3D interacting hands datasets and show the benefit of it. Our Re:InterHand is available in https://mks0601.github.io/ReInterHand/

A Dataset of Relighted 3D Interacting Hands
Gyeongsik Moon mks0601 @meta. com
Shunsuke Saito shunsukesaito@meta.. com
Weipeng Xu
xuweipengdmetaa.cm com
Rohan Joshi rohanjoshi wwtpnnjsssimmmtt...m com
Julia Buffalini jbuffalini@mete
Harley Bellan harleybellandmmeta.. com
Nicholas Rosen ni aicholasrosendmeta. com
Jesse Richardson jesserichardson com
Malloric Mize
mallori emize@meta com
Philippe de Bree phillipedeereeemmta.ccom
Tomas Simon tsimon@meta.c com
Bo Peng bopengêmeta.co com
Shubham Garg
Kevyn McPhail kmcphail @meta. com
Takaaki Shiratori tshiratoris @meta com
ssgarg@meta.com
Meta Reality Labs Research
Abstract
The two-hand interaction is one of the most challenging signals to analyze due to the self-similarity, complicated articulations, and occlusions of hands. Although several datasets have been proposed for the two-hand interaction analysis, all of them do not achieve 1) diverse and realistic image appearances and 2) diverse and large-scale groundtruth (GT) 3D poses at the same time. In this work, we propose Re:InterHand, dataset of relighted 3D interacting hands that achieve the two goals. To this end, we employ a state-of-the-art hand relighting network with our accurately tracked two-hand 3D poses. We compare our InterHaand with existing 3D interacting hands datasets and show the benefit of it. Our Re:InterHand is available here.
Introduction
Humans often make two-hand interactions during daily conversation when interacting with objects. Self-similarity, complicated articulations, and small sizes of hands make analyzing such two-hand interactions greatly challenging. In particular, when the input of analyzing system is single image, the problem becomes much more difficult as in most cases, most of a hand is occluded by the other hand.
One fundamental direction to successfully analyze interacting hands is collecting large-scale 3D interacting hands datasets, which contain in-the-wild images and corresponding 3D groundtruth (GT). Unfortunately, this is not trivial. Due to the inherent scale and depth ambiguity, true 3D data is not obtainable from single 2D observation. In addition, a single 2D observation does provide enough information of other viewpoints, necessary for the 3D data collection. Therefore, there have been three alternative approaches collect 3D hand data.
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

Ego3DHands (Composited dataset)
RedInterHand (Ours)
(Lab dataset)
(Natural dataset)
Appearance diverse?
No
Yes
Yes
Appearance realistic?
Yes
Yes
No
Middle
Appearance
Middle
GT 3D pase diverse?
Yes
No
Yes
Yes
Figure I: Comparison of datasets from existing data capture approaches ((a), (b), and (c)) and our new Re:InterHand dataset ((d)). ITW represents in-the-wild environments.
1.1 Lab datasets
Lab datasets are captured from specially designed studios with hundreds of calibrated and synchro- nized cameras, InterHand2.66 6M [30, 29] is the most widely used 3D interacting hands dataset, and it captured from a studio with 100 calibrated and synchronized cameras. Fig. (a) shows an image example of InterHand2.6M.
Pros. They provide large-scale, diverse, and accurate GT 3D poses.
Cons. Images have monotonous appearances. The figure shows that images have far and limited diversities of color, backgrounds, and illuminations compared to those of in-the-wild images.
1.2 Natural datasets
Natural datasets, such HIC [52] and RGB2Hands [53], are captured from daily environments with a much smaller number of cameras, for example, single RGBD camera. Fig. (b) shows an image example of HIC.
Pros. As the figure shows, the image appearance is close to those of tetthh-wile images.
Cons. The diversity and scale of the dataset is limited. Although the capture setup is much lighter than that of lab datasets, bringing the setup and capturing at diverse places is not easy, which makes appearance diversity limited (e.g., in front of desks). As only a few cameras are used, such datasets could not provide accurate annotations for complicated interacting hands. Therefore, they provide simple poses.
1.3 Composited datasets
Composited datasets, such as Ego3DHands [23], are composition of hand images with random background images. The purpose of the composition enhance the appearance diversity of lab images or synthesized images. Fig. (c) shows an example of it.
They often have accurate and diverse GT 3D poses the composition is performed on lab datasets or synthesized datasets.
Pros.
Cons.
The figure shows that its image appearances are not realistic due to the light inconsistency between foreground and background.
1.4 The proposed Re:InterHand dataset
All the existing three approaches have their own limitations. this work, we propose Re:InterHand dataset, which complements all three existing dataset collection approaches. Fig, (d) shows an

---

Table Comparison of hand datasets that provide GT 3D poses. To count the number of images, we consider images from different viewpoints at the same time step as different ones.
Image
# #o
# of images
Two-hand interactions
Datasets
GT
appearance
subjects
Objeet+Ohject [44]
Natural
3D fingertip coord.
3K
No
STB [59]
Natural
3D joint coord.
36K
No
EgoDexter [33]
Natural
3D fingertip coord.
3K
No
RHD [61]
Composite
3D joint coord.
20
44K
No
PanopticStudio [43]
Lab
3D joint coord.
N/A
15K
No
FPHA [9]
Natural
3D joint coord. + 3D obj.
105K
No
GANerated [31]
Composite
3D joint coord.
N/A
264K
No
FreiHAND [62]
Natural
MANO MANO obj.
32
134K
No
ObMan [15]
Composite
150K
No
EHF [39]
Lab
SMPL-X
100
No
HO3D [13]
Natural
3D joint coord. + 3D obj.
10
78K
No
ContactPose [3]
Lab
3D joint coord. 3D obj.
50
2.9M
No
HUMBI [56, 55]
Lab
MANO
453
24M
No
DexYCB [4]
Natural
MANO 3D obj.
10
582K
No
AGORA [38]
Realistic
SMPL--
350
19K
No
DART [8]
Composite
MANO
N/A
787K
No
BlurHand [34]
Lab
MANO
156K
No
H20 [21]
Natural
MANO obj.
571K
Weak
Assembly101 [42]
Natural
3D joint coord. + action labels
53
111M
Weak
AssemblyHands [35]
Natural
3D joint coord.
3M
Weak
ARCTIC [7]
Lab
MANO SMPL-X + 3D obj.
10
2.1M
Weak
HIC [52]
Natural
MANO
36K
Strong
RGB2Hands [53]
Natural
3D joint coord. fingertips
1K
Strong
InterHand2.66 6M [30]
Lab
MANO
27
2.6M
Strong
Ego3DHands [23]
Composited
3D joint coord. masks
55K
Strong
Re:InterHand (Ours)
Realistic
MANO + masks
10
1.5M
Strong
image example of our Re:InterHand dataset. Our dataset is constructed by rendering 3D hands with accurately tracked 3D poses and relighting it with diverse environment maps. By using accurately tracked 3D poses from our multi-camera studio, we could secure diverse GT 3D poses. For the relighting, we employ a state-of-the-art hand relighting network [17], which provides diverse and realistic image appearances. The figure shows that our rendered data has close appearances compared to those of in-the-wild images.
2 Related works
3D hand datasets. Tab. shows comparisons of various 3D hand datasets. Motivated by the Kinect device, early datasets comprise depthmaps [49, 51, 47, 57, 44]. For more practical applications without requiring depth cameras, RGB-based datasets have been introduced. STB [59] includes sequences with simple hand poses. HIC [52] is one of the earliest approaches address two-hand interactions. RHD [61] consists of synthetically rendered images using commercial software and composited with web-crawled background images. EgoDexter [33] includes sequences with simple hand-object interactions. Panoptic Studio [43] is captured from a specially designed dome, and contains whole-body humans. FPHA [9] includes hand sequences captured from first-person viewpoints, GANerated [31] is synthetically generated using generative adversarial networks and composited with background images. EHF [39] a small-scale dataset captured from a multi-camera studio. It includes a whole- booee-bdyy performance of single subject. ObMan [15] includes simple hand-object interactions. background images. FreiHAND [62] is captured with portable multi-camera setup in various places. consists natural images and composited images. Mueller al. [32] introduced synthetic depth map dataset of interacting two hands. YT3D [20] includes web-crawled videos and 3D pseudo-GT of hands. NeuralAnnot [29] introduced 3D pseudo-GT of hands on MSCOCO [24, 18] dataset. Both YT3D and NeuralAnnot fit a 3D hand model [40] to 2D joint coordinates to obtain 3D pseudo-GT.
synthetically rendered using commercial software and composited with
3

---

Aekrngg shartea diitance betwwan har
eoncennt cumple
3:54
ADDRA
N20
ARCTIC
Inberaand Roi moretterandd
ACOMA 100
ARCTIC
IBC
Interitend Rerinterttend
Figure 2: Left: HIC [52], IMterHand2.6M [30], and our Re:InterHand have short distance between two hands. Right: InterHand2.66 6M [30] and our Re:InterHand have many samples where two hands in contact. We count sample contacting if the shortest distance between the vertices of two hands is smaller than 3 mm. We exclude samples of ARCTIC [7] whose distance between two hands is longer than meter.
They mostly contain single-hand 3D pseudo-GT without 3D relative translation between two hands due to depth and scale ambiguity. HO3D [13] includes 3D hands interacting with various types of objects. RGB2Hand [53] introduced small-scale 3D interacting hands dataset with 3D joint coordinate annotations without fingertips. InterHand2.66, [30] is large-scale 3D interacting hands dataset, captured from a specially designed multi-camera studio. ContactPose [3] contains sequences of 3D hands and contact maps, generated from hand-object interactions. HUMBI [56, 55] is a large- scale dataset that provides 3D whole-body annotations, captured from a specially designed multi- camera studio. DexYCB [4] includes large-scale 3D hands interacting with various types of objects.
Ego3DHands [23] is a composition with random back-
ground images and rendered two-hand images. H2O [21] contains two hands interacting with objects. AGORA [38] rendered with 3D scans of people and scenes. Like our Re:InterHand dataset, AGORA considers light consistency between foreground and background, which makes their image appearances realistic. Ego4D [1]] includes a huge amount of first-person viewpoint videos; however, does not provide 3D hand annotations. DART [8] contains ren- dered images of a single hand with accessories and their texture map. dipha-blender with background images from MSCOCO [24]. Assembly 101 [42] contains large-scale videos of 3D hands assembling several objects. Assem- blyHands [35] improved Assembly100 [42] with a better annotation pipeline. ARCTIC [7] includes 3D hands and whole-body annotation with 3D objects. BlurHand [34] is made from subset of InterHand2.66 [30]. It includes blurred hand images and corresponding GT 3D hands.
4cantermand lours)
Interffen22 ON
Figure 3: t-SNE of two-hands" 3D pose of our Re:InterHand, Inter- Hand2.6M [30]. and HIC [52].
Although there have been many 3D hands datasets introduced, there is small number of datasets with
strong two-hand interactions [52, 53, 30, 23]. Among them, InterHand2.6MM [30] and HIC [52] are widely used as RGB2Hands [53] have no 3D fingertip annotations, and images of Ego3DHands [23] not photorealistic. Some datasets [61, 43, 39, 20, 3, 56, 55, 21, 38, 29, 42, 35, 34] have two-hand annotations; however, they have weak interactions between hands. Fig. 2 shows that only HIC [52]. InterHand2.6M [30], and our Re:InterHand have a short distance between two hands and meaningful ratio of contacting samples. Unfortunately. none of such two-hand datasets has achieved two goals at the same time: 1) rich and realistic image appearances and 2) accurate and diverse GT 3D poses of interacting hands. Our Re:InterHand is the first dataset that achieves the two goals. In addition, Fig. shows that our IntenHaHand has much more diverse 3D interacting hand poses than InterHand2 6. [30], and HIC [52].

---

Capture stage
Multi-view images
scan
2D/3D joint coord.
Relight stage
Relighting network
3D hand model fit
Env.
Relighted images
Figure 4: The overall pipeline of constructing our Re:InterHand dataset.
3D interacting hands recovery. Due the absence of large-scale datasets, early works [36, 52, 50, 32, 53] are based on a fitting framework, which fits 3D hand models to geometric observations, such RGBD sequence [36], hand segmentation map [32], and dense matching map [53]. Inter- Hand2.6M 6M [30, 29] motivated many pereesion-bbssed methods [41, 58, 22, 14, 5, 6, 19, 25, 26]. Such regression-basee methods outperform the above fitting-based approaches while running real-time. Li et al. [22] introduced Transformer-based network with the cross-attention between right and left hands. Moon [26] presented 3D interacting hands recovery network that addresses the domain gap between multi-camera datasets and in-the-wild datasets, which results in robust performance on in-the-win tthe-widd images.
Relighting humans. Several works [54, 46, 60, 60] are proposed to relight faces and bodies; however, these models not animatable. To enable relighting with animation, Bi al. [2] presented deep relightable appearance model for facial avatars. DART [8] provides a dataset of relighted hands; however, their images are not photorealistic as they do not consider light consistency between foreground and background. Iwase et al. [17] introduced an efficient neural relighting system for photorealistic hand relighting using student-teacher framework and feature-based relighting [37]. We use the relighting system of Iwase et al. [17] due to their high-quality results and rendering efficiency.
3 Dataset construction
Fig. 4 shows the overall pipeline for the construction of our dataset. It consists of two stages: capture and relight.
3.1 Capture stage
The capture stage captures hand data from our multi-camera studio. We capture data from 10 subjects, shown in Fig. 5. Two types of sequences, peak poses and range of motion, captured following InterHand2.0r 6M [30]. The peak pose is sequence, which includes a transition from a neutral pose to a pre-defined pose and then transition back to the neutral pose. The purpose of the peak pose is to capture as diverse poses possible including extreme poses and maximal finger bent. The range of motion a sequence, which includes natural hand motion driven with minimal instructions, such as waving hands as if friends are coming over. In this way, we could capture both 1) diverse poses from the peak pose sequences and 2) natural hand motion from the range of motion sequences. We provide more image and pose examples of our dataset in the supplementary material.

---

viewplll 3rd cal--prsson viewpoine
egocentric viewpoints
Figure 5: Each column shows images of a subject of our Re:InterHand dataset. For each column, the top image with the neutral pose from the capture stage, and the remaining images with captured poses are from the relight stage.
Capture studio. Our capture studio has 469 lights and 170 calibrated synchronized cameras. All cameras lie on the front, side, and top hemispheres of the hand and are placed at a distance of about one meter from
Images are captured with 4096 x 2668 pixels at 90 frames per second (fps). Following Bi et al. [2], we interleave fully lit frames and partially lit frames at every 3 frames. The capture stage only uses fully lit frames, and the relight stage uses partially lit frames to train the relighting network.
2D joint coordinates and 3D scans. We process the raw video data by performing 2D joint detection [45] and 3D scan [12]. The 2D joint detector is trained on our held-out manually annotated dataset, which includes 900K images with rotation center coordinates of hand joints, where our manual annotation tool is similar to that of Moon et al. [30]. Our 2D joint detector has an error of 2.5 pixels in a 1024 X 667 image space.
3D joint coordinates. InterHand2266 [30] triangulated detected multi-view 2D joint coordinates with the RANSAC algorithm. We found that their approach suffers from temporally inconsistent results as the triangulation does not take into account the similarity between close frames. For example, some joints could have inconsistent semantic positions across viewpoints due the failure of the 2D joint detector. In this case, triangulated 3D coordinates of such joints could be very different between close frames if selected viewpoints by RANSAC different. Instead triangulation, we train a 3D joint detection network, which takes a voxelized 3D scan of hands and is supervised with multi-view 2D joint coordinates. Our network produces much more temporally consistent and smooth results as inputs of close frames (i.e., voxelized 3D scans) are almost the same.
The network designed with V2V-PoseNet [27], state-of-the-art 3D joint detection network from voxelized hands. First, we make two volumes from 3D scans by making 3D bounding boxes around the mean of initially obtained left and right hands' 3D joint coordinates, where the initial obtained with the RANSAC algorithm. Then, we voxelize 3D scans around each 3D bounding box to (96, 96, 96) resolution. The voxelized 3D scans are passed to the V2V-PoseNet. which consists of stacked 3D convolutional layers. We perform soft-argmax [48] to the output of the V2V-PoseNet, which produces 3D joint coordinates in differentiable way. The obtained 3D joint coordinates are supervised with multi-view 2D joint coordinates by projecting the 3D ones to each viewpoint and calculating L1 distance from the 2D ones. We train V2V-PoseNet on all frames, which takes day, and test it on the same frames to obtain 3D joint coordinates of them. Our obtained 3D joint coordinates have an error of 2.0 mm. The errors are measured against our held-out human-annotated set.

---

3D hand model fitting, We additionally obtain 3D meshes of hands as 1) they provide useful surface information that does not exist in the 3D joint coordinates and 2) they are inputs of the relighting network [17]. To this end, we fit 3D hand models, such as MANO [40]. to the obtained 3D joint coordinates and 3D scans of the above using NeuralAnnot [29]. The 3D hand model is parametric model that produces 3D hand meshes from 3D pose and identity (ID) codes. The 3D pose represents 3D joint angles, and ID codes determine 3D hand shape, such as thickness. in the zero pose.
trangulated
(6) 3D hand model from V2V-PaseNet (Curs)
NeuralAnnot takes single image and 3D joint coordinates as inputs and outputs 3D pose and ID codes, used to drive 3D hand models. We use the network architecture of Pose2Pose [28] for NeuralAnnot. The 3D pose and ID codes are supervised with the 3D joint coordinates after performing forward kinematics. Also, 3D meshes from the 3D pose and ID codes are supervised with 3D scans by minimizing the closest distance between 3D meshes and 3D scans. Several regularizers, such as 1) L2 regularizers to 3D pose and ID codes, which prevents extreme meshes, and 2) a collision avoidance regularizer are applied as well. We separately train NeuralAnnot for each subject, and the ID code is directly optimized, not regressed from the input image and 3D joint coordinates. In this way, 3D hands from the same subject have consistent ID code. Training Nnno takes less than hour for each capture. After training NeuralAnnor we test it on the training set and manually inspect all frames. Frames with wrong fitting results are excluded for the following relight stage. Fig. 6 shows that it produces temporally consistent results. We checked that the MANO meshes from NeuralAnnot have 1.3 mm errors from the 3D scans without any translationrrotationnsccll alignments.
Figure 6: Comparison of 3D human model fits from (a) triangulation of In- terHand2.6N [30] and (b) our V2V- PoseNet. The three frames are con- secutive ones, and the time difference between near frames is 0.02 seconds. Given the very short time difference be- tween frames, the three frames should have almost the same 3D hands. (a) not only suffers from the collisions but also suffers from temporal inconsistency be- tween very close frames. On the other hand, (b) does not suffer from the colli- sions and achieves temporal consistency between close frames.
3.2 Relight stage
After capturing data in the above capture stage, we train relighting network [17] for each subject following their original training strategy. Following them, we train the relighting networks on single- hand data as 3D hand model fittings are more accurate for the single-hand data than the two-hand data, which makes training the relighting network more stable. Please note that the single-hand data to train the relighting networks are also obtained from NeuralAnnot by training and testing it on single-hand captures. For more details, please refer to Twase et al. [17]. After training the relighting networks, we use the 3D poses from NeuralAnnot [29] of the above capture stage to render two hands with specified camera parameters. For illuminations, we use 2144 high-resolution environment maps of Gardner al. [10].
Dataset release
Our Intentarandd dataset includes 1) relighted images, 2) non-binary masks, and 3) 3D hand model fittings, as shown in Fig. 7. The relighted images and non-binary foreground masks are from Sec. 3.2, and 3D hand model fittings are from Sec. 3.1. Out 10 captures, we split 7 captures for the training set and the remaining 3 captures for the testing set.
Relighted images. To render relighted images, we first sample cameras out of our 170 cameras for each capture to make overall rendering faster and remove redundancy. To sample cameras, we sum 2D joint detection confidence from Sec. 3.1 for each camera. Then, we pick the top 50 cameras based on the sum of confidence values. In this way, we can exclude cameras where hands are almost not visible. The farthest iterative sampling algorithm samples N cameras from the selected cameras based on the camera positions to obtain as diverse viewpoints possible. For the frame-base
7

---

3rd ard-persoo newpppnn
egocentric viewpoints
(b)
(b)
Figure 7: (a) relighted rendered images, (b) masked images, and (c) MANO fitting overlay.
research, we downsample captures at 5 fps and set N = 20. Then, we render images with a different environment map for each frame, which results in 493K images. Also, for the video-based research, we set N 5 and render images at 30 fps with different environment map for each segment, which results in 739K images. For both frame-based and video-based split, images with the same frame index and different viewpoints are rendered from the shared environment map in a multi-view consistent way.

---

Table 2: RRVE comparison of InterWild [26] trained on different data including variants of our
dataset. We use the 3rd-person viewpoint split of our Re:InterHand.
Testing sets
Training sets
InterHand2..05
HIC
Re:InterHand
InterHand2.6M MSCOCO
19.74
23.59
37.59
+ Capture stage
19.14
23.09
34.23
+ Capture stage and Composite
19.50
24.37
31.30
+ Capture stage and Composite (w. AdaIn [16])
19.44
24.83
27.96
+ Capture stage and Relight stage (Ours)
19.40
21.36
20.07
(b) Capture stage +
(c) Capture stage + Composite (w. Adain)
(d) Capture stage Relight stage (Ours)
(a) Capture stage
Composite
Figure Image examples of datasets constructed in Tab.
One advantage of our approach is that we can render images with any novel camera parameters. In addition to the above pre-defined 3rd-person viewpoints, we also render relighted images from random egocentric viewpoints to contribute our Re:InterHand to the egocentric 3D hand community. To this end, we first manually put a reference camera in the middle of two eyes using 3D scans that include both hands and face. The orientation of the reference camera is set to see the center of the hands. Then, for each frame, we randomize 3D camera positions within 20 cm of 3D box around the reference camera. We also randomize the 3D orientation of the camera by applying [-30", 30"] pitch, yaw, and roll. The principal point is set to the image center, and the focal length is randomized from [0.7, 1.8] times the image size. To simulate the fisheye cameras, often used for egocentric viewpoints, we randomize distortion the fisheye cameras by pre-defined mean and standard deviation. For the frame-base based research, render images with different environment map for each frame at 30 fps, which results in 148K images. Also, for the video-based research, we render images with a different environment map for each segment 30 fps, which results in 148K images.
For each peak pose sequence, we exclude frames at the first and last segment whose velocity of hands is less than threshold to remove many neutral pose frames, Both 3rd-person and egocentric viewpoints images are rendered in 1K resolution.
Non-binary masks. We provide non non-binary masks, obtained from the relight stage. The non-binary mask different from binary masks rendered from MANO fittings as the non-binary ones are perfectly aligned with images including detailed silhouettes, such as nail and muscle bulging.
3D hand model fittings. We provide MANO [40] fitting it is the most widely used 3D hand model in the community. Also, we provide the 3D hand model fittings, used to render relighted images.
Experiments
For all experiments, we report right hand-relative vertex error (RRVE), Euclidean distance (mm)
between estimated and GT 3D meshes of two hands after aligning translation of the right hand's root joint (i.e., wrist). Note that the most widely used metric of previous works [58, 22, 26] (MPVPE) is calculated after aligning the translation of the right and left hand separately; hence, their MPVPE does not consider relative position between two hands, while our RRVE does. For the 3rd-person viewpoint experiments, we report RRVE on the test split of InterHand2.cr 6M (H) [30], HIC [52], and the test split of our InteeHaaand, For the egocentric viewpoint experiments, we report RRVE on the test split of our InterHaand after training methods on the training split of it. For all experiments,

---

Table 3: Benchmark of ffate-of-th--eee methods with the RRVE metric. Methods with are tested with GT hand boxes. Methods with indicate that they are trained additionally on Re:InterHand. We use the 3rd-person viewpoint split of our Re:InterHand
Testing sets
Methods
InterHand2266M
HIC
Re:InterHand
Hand [22]
19.96
67.11
52.91
InterWild [26]
19.74
23.59
37.59
InterWild [26]*
19.40
21.36
20.07
Table 4: Benchmark on the egocentric viewpoint split of our Re:InterHand.
Methods
RRVE
InterWild [26]
28.89
the frame-based split of Re:InterHand is used. For all datasets, the errors are calculated only for
two-hand samples.
Effectiveness of the relight stage. Tab. 2 shows the effectiveness of the relight stage. noteworthy that relight stage greatly reduces the error of HIC, which consists of real and natural images. Please note that images of HIC are entirely novel ones as they are only used for the testing purpose. Our relight stage also significantly reduces the test error our Re:InterHand test set while slightly reducing errors on InterHand2.66M of lab images, the data from the capture studio (the second row of the table) reduces the error InterHand2.66 the most. However, it could not improve the test results on HIC with real and natural images image appearances
As both InterHand2.66
and data from the capture stage consist
far from those of real images, shown in Fig. (a). The variants with composition (the third and fourth rows) make the performance on HIC of the baseline (the first row) worse. We think the reason is that their images have inconsistent light between foreground and background, as shown in Fig. 8 (b) and (c). For more harmonic images, we applied AdaIn [16] raw RGB pixels of the foreground to make them follow distributions of the background pixels. Unfortunately, as it is not aware of the reflections, it often changes the colors of hands to unrealistic colors without preserving skin colors and only changing lights, which results in performance degradation on HIC with real and natural image appearances.
Benchmark. Tab. 3 and provide benchmark results with IntagHand [22] and InterWild [26]. 3D interacting hands recovery methods. We use their official checkpoints, and GT hand boxes are used for IntagHand assumes them.
6 Conclusion
Summary. We present Re:InterHand dataset, which provides images with highly realistic and diverse appearances of interacting hands and their corresponding GT 3D hands. To this end, our accurately tracked 3D poses, the state-of-the- art relighting network [17]. and number of high-resolution environment maps are used. We hope our dataset can make the community one step closer to the interacting hands recovery in the wild.
Limitations. As Fig. 5 shows, our rendered images have cut the forearm area. This is because our relighting network only takes a 3D hand geometry, not a whole-body one. We think this is not severe issue most 3D hand analysis systems take cropped hand images followed by hand detectors, where hand detectors can be trained on large-scale real datasets only with 2D annotations. Also, we observed that there are sometimes artifacts in the relighted images. This because the relighting network is trained on single-hand data and tested on two-hand data, which sometimes results in pose generalization failure. We expect better relighting network could alleviate this issue.
10

---

References
[1] Luca Ballan, Aparna Taneja, Jürgen Gall, Luc Van Gool, and Mare Pollefeys. Motion capture of hands in action using discriminative salient points. In ECCV, 2012.
Sai Bi, Stephen Lombardi, Shunsuke Saito, Tomas Simon, Shih-EEn Wei, Kevyn Mcphail, Ravi Ramamoor- thi, Yaser Sheikh, and Jason Saragih. Deep relightable appearance models for animatable faces, ACM TOG, 2021.
[3] Samarth Brahmbhatt, Chengcheng Tang. Christopher D. Twigg, Charles C. Kemp, and James Hays. ContactPose: A dataset of grasps with object contact and hand pose. In ECCV, August 2020.
[4] Yu-Wei Chao, Wei Yang, Yu Xiang, Pavlo Molchanov, Ankur Handa, Jonathan Tremblay, Yashraj S Narang, Karl Van Wyk, Umar lqbal, Stan Birchfield, et al. Dex YCB: A benchmark for capturing hand grasping of objects. In CVPR, 2021.
[5] Xinhan Di and Pengqian Yu. LWA-HAND: Lightweight attention hand for interacting hand reconstruction. In ECCVW, 2022.
[6] Zicong Fan, Adrian Spurr, Muhammed Kocabas, Siyu Tang. Michael J Black, and Otmar Hilliges. Learning to disambiguate strongly interacting hands via probabilistic per-pixel part segmentation. In 3DV, 2021.
[7] Zicong Fan, Omid Taheri, Dimitrios Tzionas, Muhammed Kocabas, Manuel Kaufmann, Michael J. Black, and Otmar Hilliges. ARCTIC: A dataset for dexterous bimanual hand-object manipulation. In CVPR, 2023.
[8] Daiheng Gao, Yuliang Xiu, Kailin Li, Lixin Yang, Feng Wang, Peng Zhang, Bang Zhang, Cewu Lu, and Ping Tan. DART: Articulated hand model with diverse accessories and rich textures. In NeurIPS Datasets and Benchmarks Track, 2022.
[9] Guillermo
Shanxin Yuan, Seungryul Back, and Kyun Kim. First-person hand action benchmark with RGB-D videos and 3D hand pose annotations. In CVPR, 2018.
[10] Mare-André Gardner, Kalyan Sunkavalli, Ersin Yumer, Xiaohui Shen, Emiliano Gambaretto, Christian Gagné, and Jean-François Lalonde. Learning to predict indoor illumination from a single image. ACM TOG, 2017.
[11] Kristen Grauman, Andrew Westbury, Eugene Byrne Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4D: Around the world in 3,000 hours of egocentric video. In CVPR, 2022.
[12] Kaiwen Guo, Peter Lincoln, Philip Davidson, Jay Busch, Xueming Yu, Matt Whalen, Geoff Harvey, Sergio Orts-Eseolano, Rohit Pandey, Jason Dourgarian, al. The relightahles: Volumetric performance capture of humans with realistic relighting, ACM TOG, 2019.
[13] Shreyas Hampali, Mahdi Rad, Markus Oberweger, and Vincent Lepetit. Honnotate: A method for 3D annotation of hand and object poses. CVPR, 2020.
[14] Shreyas Hampali, Sayan Deb Sarkar, Mahdi Rad, and Vincent Lepetit. Keypoint transformer: Solving joint identification challenging hands and object interactions for accurate 3D pose estimation. In CVPR, 2022.
[15] Yana Hasson, Gül Varol, Dimitris Tzionas, Igor Kalevatykh, Michael Black, Ivan Laptev, and Cordelia Schmid, Learning joint reconstruction of hands and manipulated objects. In CVPR, 2019.
[16] Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In ICCV, 2017.
[17] Shun Twase, Shunsuke Saito, Tomas Simon, Stephen I Lombardi, Timur Bagautdinov, Rohan Joshi, Fabian Prada, Takaaki Shiratori, Yaser Sheikh, and Jason Saragih. RelighableHHands: Efficient neural relighting of articulated hand models. In CVPR, 2023.
[18] Sheng Jin, Lumin Xu, Jin Xu, Can Wang, Wentao Liu, Chen Qian, Wanli Ouyang, and Ping Luo. Whole- body human pose estimation in the wild. In ECCV, 2020.
[19] Dong Uk Kim, Kwang In Kim, and Scungryul Back. End-to-end detection and pose estimation of two interacting hands. In /CCV, 2021.
[20] Dominik Kulon, Riza Alp Guler, lasonas Kokkinos, Michael M. Bronstein, and Stefanos Zafeiriou. hand reconstruction in the wild. In CVPR, 2020.
11

---

[21] Taein Kwon, Bugra Tekin, Jan Stühmer, Federica Bogo, and Marc Pollefeys. H2O: Two hands manipulating objects for first person interaction recognition. In ICCV, 2021.
[22] Mengcheng Li. Liang An. Hongwen Zhang. Lianpeng Wu, Feng Chen, Tao Yu, and Yebin Liu. Interacting attention graph for single image two-hand reconstruction. In CVPR, 2022.
[23] Fanqing Lin, Connor Wilhelm, and Tony Martinez. Two-hand global 3D pose estimation using monocular RGB. In WACV, 2021.
[24] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In ECCV, 2014.
[25] Hao Meng, Sheng Jin, Wentao Liu, Chen Qian, Mengxiang Lin, Wanli Ouyang, and Ping Luo. 3D interacting hand pose estimation by hand de-occlusion and removal. In ECCV, 2022.
[26] Gyeongsik Moon. Bringing inputs to shared domains for 3D interacting hands recovery the wild. In CVPR, 2023.
[27] Gycongsik Moon, Ju Yong Chang, and Kyoung Mu Lec. V2V-PoseNet: Voxel-to-voxel prediction network for accurate 3D hand and human pose estimation from a single depth map. In CVPR, 2018.
[28] Gyeongsik Moon, Hongsuk Choi, and Kyoung Mu Lee. Accurate 3D hand pose estimation for whole-hody 3D human mesh estimation. CVPRW, 2022.
[29] Gyeongsik Moon, Hongsuk Choi, and Kyoung Mu Lee. NeuralAnnot: Neural annotator for 3D human mesh training sets, In CVPRW, 2022.
[30] Gycongsik Moon, Shoou-1 Yu, He Wen, Takaaki Shiratori, and Kyoung Mu Lee. InterHand2.6N :M::: A dataset and baseline for 3D interacting hand pose estimation from single RGB image. In ECCV, 2020.
[31] Franziska Mueller, Florian Bernard, Oleksandr Sonnychenko, Dushyant Mehta, Srinath Sridhar, Dan Casas. and Christian Theobalt. GANerated hands for real-time 3D hand tracking from monocular RGB. In CVPR, 2018.
[32] Franziska Mueller, Micah Davis, Florian Bernard, Oleksandr Sotnychenko, Mickeal Verschoor, Miguel A Otaduy, Dan Casas, and Christian Theobalt. Real-time pose and shape reconstruction of two interacting hands with a single depth camera. ACM TOG, 2019.
[33] Franziska Mueller, Dushyant Mehta, Oleksandr Sotnychenko, Srinath Sridhar, Dan Casas, and Christian Theobalt. Real-time hand tracking under occlusion from an egocentrie RGB-D sensor. In ICCV, 2017.
[34] Ycounguk Oh, JoonKyu Park, Jacha Kim, Gyeongsik Moon, and Kyoung Mu Lee. Recovering 3D hand mesh sequence from a single blurry image: new dataset and temporal unfolding. In CVPR, 2023.
[35] Takehiko Ohkawa, Kun He, Fadime Sener, Tomas Hodan, Luan Tran, and Cem Keskin. AssemblyHands: towards egocentric activity understanding via 3D hand pose estimation. In CVPR, 2023.
[36] lasonas Oikonomidis, Nikolaos Kyriazis, and Antonis A Argyros. Tracking the articulated motion of two strongly interacting hands. In CVPR, 2012.
[37] Rohit Pandey, Sergio Orts Escolano, Chloe Legendre, Christian Haene, Sofien Bouaziz, Christoph Rhemann, Paul Debevec, and Sean Fanello. Total relighting: learning to relight portraits for background replacement. ACM TOG, 2021.
[38] Priyanka Patel. Chun-Hao P Huang, Joachim Tesch, David T Hoffmann, Shashank Tripathi, and Michael J Black. AGORA: Avatars in geography optimized for regression analysis. In CVPR, 2021.
[39] Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed AA Osman, Dimitrios Tzionas, and Michael J Black. Expressive body capture: 3D hands, face, and body from single image. In CVPR, 2019.
[40] Javier Romero, Dimitrios Tzionas, and Michael Black, Embodied Hands: Modeling and capturing hands and bodies together. ACM TOG, 2017.
[41] Yu Rong, Jingho Wang, Ziwei Liu, and Chen Change Loy. Monocular 3D reconstruction of interacting hands via aollision-aware factorized refinements. In 3DV, 2021.
[42] Fadime Sener. Dibyadip Chatterjee, Daniel Shelepov, Kun He, Dipika Singhania, Robert Wang. and Angela Yao. Assembly100 101: A large-scale multi-view video dataset for understanding procedural activities. In CVPR, 2022.
12

---

[43] Tomas Simon, Hanbyul Joo, lain Matthews. and Yaser Sheikh. Hand keypoint detection in single images using multiview bootstrapping. In CVPR, 2017.
[44] Srinath Sridhar, Franziska Mueller, Michael Zollhöfer, Dan Casas, Antti Oulasvirta, and Christian Theobalt. Real-time joint tracking of a hand manipulating an object from RGB-D input. In ECCV, 2016.
[45] Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang. Deep high-resolution representation learning for human pose estimation. In CVPR, 2019.
[46] Tiancheng Sun, Jonathan T Barron, Yun-Ta Tsai, Zexiang Xu, Xueming Yu, Graham Fyffe, Christoph Rhemann, Jay Busch, Paul E Debevec, and Ravi Ramamoorthi. Single image portrait relighting. ACM TOG, 2019.
[47] Xiao Sun, Yichen Wei, Shuang Liang, Xiaoou Tang, and Jian Sun. Cascaded hand pose regression. In CVPR, 2015.
[48] Xiao Sun, Bin Xiao, Fangyin Wei, Shuang Liang, and Yichen Wei. Integral human pose regression. In ECCV, 2018.
[49] Danhang Tang. Hyung Jin Chang, Alykhan Tejani, and Tac-Kyun Kim. Latent regression forest: Structured estimation of 3D articulated hand posture. In CVPR. 2014.
[50] Jonathan Taylor, Lucas Bordeaux, Thomas Cashman, Bob Corish, Cem Keskin, Toby Sharp, Eduardo Soto, David Sweeney, Julien Valentin, Benjamin Luff, Efficient and precise interactive hand tracking through joint, continuous optimization of pose and correspondences. ACM TOG, 2016.
[51] Jonathan Tompson, Murphy Stein, Yann Lecun, and Ken Perlin. Real-time continuous pose recovery of human hands using convolutional networks. ACM TOG, 2014.
[52] Dimitrios Tzionas, Luca Ballan, Abhilash Srikantha, Pablo Aponte, Mare Pollefeys, and Juergen Gall. Capturing hands in action using discriminative salient points and physics simulation. IJCV, 2016.
[53] Jiay Wang. Franziska Mueller, Florian Bernard, Suzanne Sorli, Oleksandr Sotnychenko, Neng Qian. Miguel A Otaduy, Dan Casas, and Christian Theobalt. RGB2Hands: real-time tracking of 3D hand interactions from monocular RGB video. ACM TOG, 2020.
[54] Shugo Yamaguchi, Shunsuke Saito, Koki Nagano, Yajie Zhao, Weikai Chen, Kyle Olszewski, Shigeo Morishima. and Hao Li. High-fidelity facial reflectance and geometry inference from unconstrained image. ACM TOG, 2018
[55] Jae Shin Yoon, Zhixuan Yu, Jaesik Park, and Hyun Soo Park. HUMBI: A large multiview dataset of human body expressions and benchmark challenge. TPAMI, 2021.
[56] Zhixuan Yu, Jae Shin Yoon, In Kyu Lee, Prashanth Venkatesh, Jaesik Park, Jihun Yu, and Hyun Soo Park. HUMBI: A large multiview dataset of human body expressions. In CVPR, 2020.
[57] Shanxin Yuan, Qi Ye, Bjom Stenger, Siddhant Jain, and Tae- Tae-Kyun Kim. Bighand2.2M benchmark: Hand pose dataset and state of the art analysis. In CVPR, 2017.
[58] Baowen Zhang, Yangang Wang, Xiaoming Deng, Yinda Zhang, Ping Tan, Cuixia Ma, and Hongan Wang. Interacting two-hand 3D pose and shape reconstruction from single color image. In ICCV, 2021.
[59] Jiawei Zhang. Jianbo Jiao, Mingliang Chen, Liangqiong Qu, Xiaobin Xu, and Qingxiong Yang. 3D hand pose tracking and estimation using stereo matching. arXiv preprint 11110:161..02214. 2016.
[60] Xiuming Zhang, Sean Fanello, Yun-Ta Tsai, Tiancheng Sun, Tianfan Xue, Rohit Pandey, Sergio Orts- Escolano, Philip Davidson, Christoph Rhemann, Paul Dehevec, et al. Neural light transport for relighting and view synthesis. ACM TOG, 2021.
[61] Christian Zimmermann and Thomas Brox. Learning to estimate 3D hand pose from single RGB images. In ICCV, 2017.
[62] Christian Zimmermann, Duygu Ceylan, Jimei Yang, Bryan Russell, Max Argus, and Thomas Brox FreiHand: A dataset for markerless capture of hand pose and shape from single RGB images. ICCV, 2019.
13


Reviews:
[{"review_id": "wSqVSqwChn", "reviewer": "Reviewer_AoTd", "summary": "", "questions": "", "limitations": "I think this work does not have negative societal impacts.", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "The submission has several strengths: \n\n1. The paper presents a clear and well-motivated introduction, setting the stage effectively for the research presented.\n\n2. The introduction of the methodolgy in the paper is comprehensive, and the results demonstrate the benefits of the proposed dataset on 3D interacting hands recovery.", "weaknesses": "", "quality": null, "clarity": 1, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "N8xYbRePq6", "reviewer": "Reviewer_rLCe", "summary": "", "questions": "", "limitations": "Authors have adequately addressed the limitations and potential negative societal impact of their work.", "rating": 10, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "[+] Contributing a large scale realistic dataset for interacting hands can benefit the future study for this area. (should be public released)\n\n[+] Excellent analysis of previous works, highlighting their strengths and weaknesses.\n\n[+] Providing comprehensive analysis for the dataset.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "0IilRpZphS", "reviewer": "Reviewer_vVQs", "summary": "", "questions": "", "limitations": "The quality of dataset proposed is limited; while the dataset is large in scale.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "The proposed dataset is large in scale and have relatively good appearances. ", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: This paper presents a pipeline for capturing, relighting, and re-rendering interacting hands, and utilizes this pipeline to introduce a new, large-scale, realistic synthetic dataset of interacting hands. In comparison to existing hand datasets, the proposed dataset offers close two-hand interactions, a broader diversity of poses, appearances, and illumination conditions, all at a larger scale and with reasonably high realism. Additionally, the paper demonstrates the advantages of using this dataset with a state-of-the-art hand recovery method.

Pros: The paper presents a clear motivation and a comprehensive introduction. The designed synthesis pipeline utilizes state-of-the-art techniques to re-render high-quality hand images. Consequently, the paper contributes a large-scale, high-quality interacting hands dataset with diverse poses, appearances, and illumination conditions, from which the community will benefit. Moreover, the results clearly demonstrate the advantages of this proposed dataset for 3D interacting hand recovery.

Cons: Although the dataset captures a wide diversity of appearances and poses in-the-wild setting and reasonably realistic, it is a synthetic dataset, and its quality is not as high as real images. Additionally, there is room for improvement in the explanation of the algorithms used in the pipeline and in the analysis to enhance the overall presentation.

Reviewers are in agreement that the paper is well-motivated and clear, with a well-designed pipeline and a comprehensive dataset that addresses a gap in large-scale interacting hands in real-world scenarios. Following the rebuttal, there remains a concern regarding the dataset's quality. Although the re-rendered images may not fully match the realism of real images, their quality is reasonably high, and the experiments clearly demonstrate the dataset's benefits for 3D hand recovery. This pipeline and dataset are expected to be valuable contributions to the field.
Decision: Accept (Poster)