Title: Learning Interaction-aware 3D Gaussian Splatting for One-shot Hand Avatars
Venue: NeurIPS 2024
Accepted: yes
Keywords: Gaussian splatting

interacting-hand avatar

one-shot learning
Abstract: In this paper, we propose to create animatable avatars for interacting hands with 3D Gaussian Splatting (GS) and single-image inputs. Existing GS-based methods designed for single subjects often yield unsatisfactory results due to limited input views, various hand poses, and occlusions. To address these challenges, we introduce a novel two-stage interaction-aware GS framework that exploits cross-subject hand priors and refines 3D Gaussians in interacting areas. Particularly, to handle hand variations, we disentangle the 3D presentation of hands into optimization-based identity maps and learning-based latent geometric features and neural texture maps. Learning-based features are captured by trained networks to provide reliable priors for poses, shapes, and textures, while optimization-based identity maps enable efficient one-shot fitting of out-of-distribution hands. Furthermore, we devise an interaction-aware attention module and a self-adaptive Gaussian refinement module. These modules enhance image rendering quality in areas with intra- and inter-hand interactions, overcoming the limitations of existing GS-based methods. Our proposed method is validated via extensive experiments on the large-scale InterHand2.6M dataset, and it significantly improves the state-of-the-art performance in image quality. Code and models will be released upon acceptance.

Learning Interaction-aware 3D Gaussian Splatting for One-shot Hand Avatars

Abstract

In this paper, we propose to create animatable avatars for interacting hands with 3D Gaussian Splatting (GS) and single-image inputs. Existing GS-based methods designed for single subjects often yield unsatisfactory results due to limited input views, various hand poses, and occlusions. To address these challenges, we introduce a novel two-stage interaction-aware GS framework that exploits cross-subject hand priors and refines 3D Gaussians in interacting areas. Particularly, to handle hand variations, we disentangle the 3D presentation of hands into optimization-based identity maps and learning-based latent geometric features and neural texture maps. Learning-based features are captured by trained networks to provide reliable priors for poses, shapes, and textures, while optimization-based identity maps enable efficient one-shot fitting of out-ofdistribution hands. Furthermore, we devise an interaction-aware attention module and a self-adaptive Gaussian refinement module. These modules enhance image rendering quality in areas with intra- and inter-hand interactions, overcoming the limitations of existing GS-based methods. Our proposed method is validated via extensive experiments on the large-scale InterHand2.6M dataset, and it significantly improves the state-of-the-art performance in image quality. Project Page: https://github.com/XuanHuang0/GuassianHand .

1 Introduction

Recent advancements in 3D reconstruction and differential rendering techniques have significantly improved hand avatar creation and related applications. However, creating avatars for interacting hands from a single image remains challenging. The limited input view does not provide sufficient geometry and texture information for accurate reconstruction. Moreover, intra- and inter-hand interactions exacerbate information loss and introduce complex geometric deformations.

Extensive efforts have been made to tackle these issues, as shown in Figure 2: (a) Early approaches depend on explicit parametric meshes (e.g. MANO [1]) for geometry modeling, and utilize UV map [2, 3, 4, 5], vertex color [6, 7], or image space rendering [8] for appearance. Despite the efficiency in rendering, these methods fail to achieve realistic rendering results with the coarse mesh resolution and the simple combination of hand appearance and geometry. (b) More recently,

Animatable

Avatar

Text to Hand

Single to Interacting Hands

'hand,

light skin'

'hand,

iron man'

Hand Combination

Rendering with Novel Poses and View

Meur

IPS

with the significant success of neural radiance fields (NeRF), extensive studies [9, 10, 11, 12, 13] have employed NeRF-based models for implicit modeling. These methods [10, 11, 12] usually require per-scene optimization for each new identity using densely calibrated images, which results in expensive training costs. Generalizable NeRFs [14, 15, 13, 16] get rid of per-scene training by leveraging image-aligned features to enable reconstruction from a few or even a single view. Yet their dependence on image-aligned features also limits their performance under large view or pose variations. Besides, (c) One-shot NeRF-based methods [17, 18] propose to exploit data-driven priors with condition optimizations [17] and inversions [18]. Nevertheless, these methods are not suitable for our task, as they do not include any module to detect and handle interactions. Moreover, the inversion of identity vectors used in [18] omit the spatial image structure, which not only hinders its performance but also introduces extra time consumption for fine-tuning networks.

To tackle the above issues in existing methods, we aim to create animatable avatars for interacting hands with 3D Gaussian Splatting (GS) and single-image inputs. To this end, we introduce a novel two-stage interaction-aware GS framework as shown in Figure 2 (d). We disentangle the 3D presentation of hands into (i) features that can be effectively captured by training networks in the first stage of our framework (e.g., geometric features and latent neural texture maps), and (ii) identity maps that can be optimized efficiently in the per-subject one-shot fitting stage. In this way, our method not only enables leveraging cross-subject priors with learning-based features, but also well preserves per-subject characteristics via optimizing identity maps.

2D/3D Conditions

Conditional

Generator

Rendered Image

Differentiable

Renderer

Latent

Feature Maps

Image

Encoder

Rendered Image

(a)

(b)

(c)

Supervise

Identity Vector

Partial

Modules

Partial

Modules

Differentiable Renderer

Rendered Image

Inversion

Supervise

Cross-subject

Identity Maps

Texture Conditions Neural Texture Maps

Splatting

Interaction Points

Rendered Images

IGSN

: Interaction-Aware

Training Images

Supervise

IGSN

Rendered Image

{       ,

}

Neural Texture

Target

Identity Map

Supervise

Stage-I: Learning

Disentangled Hand Priors

Stage-II: One-shot Interacting-hand Avatar Fitting

(d)

Additionally, to achieve robust reconstruction and enhance rendering quality, we devise an interactionaware attention module and a self-adaptive Gaussian refinement module. The former module identifies Gaussian points with potential intra- and inter-hand interactions to enhance their features

with attention mechanisms. This enhancement allows our GS network to better model geometric deformations and fine-grained textures caused by interactions (e.g., wrinkles). The latter module is introduced to address the limitations of the coarse geometry of parametric hand meshes, which is achieved by learning to eliminate redundant Gaussians and assigning extra Gaussians in regions with complex textures and deformations. Consequently, our method can reconstruct realistic inter-hand avatars with great flexibility for animation and editing, as shown in Figure 1.

Overall, our contributions can be summarized as follows:

· We propose a novel two-stage interaction-aware GS framework to create animatable avatars for interacting hands from single-image inputs. Our method generates high-fidelity rendering results and supports various applications. Experimental results on the large-scale Interhand2.6M dataset [19] validate the superior performance of our method compared to previous methods.

· We disentangle the 3D presentation of hands into learning-based features that can be generalized well to different subjects and identity maps that are individually optimized for each subject. This disentanglement provides us with flexible and reliable priors for poses, shapes, and textures.

· We introduce an interaction-aware attention module, which identifies intra- and inter-hand interactions and further exploits interaction context to improve rendering quality.

· We devise a Gaussian refinement module that adaptively adjusts the number and positions of 3D Gaussian, which results in rendered images of higher quality under various hand poses and shapes.

2 Related Work

One-shot Human Reconstruction . One-shot reconstruction of 3D humans is a challenging and longstanding problem due to the limited information from a single input image. To alleviate this limitation, previous works leveraged parametric models [20, 1] as coarse geometry prior. Traditional methods [21, 22, 23, 24, 25] utilized UV maps for human appearance representation. To complete the unseen texture from the single image, some approaches [25, 24] inpainted missing textures via pre-trained diffusion models. Neural Radiance Fields (NeRF) [26] have also been explored for reconstruction from sparse views [27, 15, 14, 28] or a single view [29, 16, 13, 18]. KeypointNeRF [15] encoded spatial information using 3D skeleton keypoints. SHERF [16] created 3D human avatars from a single image with hierarchical features for informative encoding. V ANeRF [13] leveraged visibility in both feature fusion and adversarial learning for single-view interacting-hand image synthesis. These methods greatly enhance the NeRF one-shot reconstruction performance. Nonetheless, generalizable NeRF [27, 30, 31, 14, 15] fails to achieve satisfactory results when the input image information is not sufficient for novel view prediction due to the under-utilization of hand priors. To overcome this issue, the pioneering research in [18] enabled one-shot single-hand avatar creation by learning data-driven hand priors which are further utilized with inversion and fitting. Compared with previous methods, our approach further disentangles the presentation of hand priors into latent geometric features, neural texture maps, and optimizable identity maps to enhance rendering quality and reduce the cost of one-shot fitting. Moreover, we introduce the interaction-aware module and self-adaptive refinement module, which helps to significantly improve the visual quality of synthesized images.

Animatable Hand Avatar . Conventional methods created hand avatars by incorporating UV textures with explicit parametric hand models. HTML [2] is the first parametric texture model of human hands which models hand appearance with several dimensions of variability. HARP [3] further introduced albedo and normal information into UV maps to represent hand appearance without any neural components. Handy [5] realistically captured high-frequency detailed texture using a GANbased texture model. However, the rendering quality of these methods is constrained by the coarse geometry and sparsity of the MANO mesh. The recent advancements in the neural radiance field have resulted in the development of approaches that utilize implicit representation for hand reconstruction. LISA [9] is the first method that employs NeRF to learn the implicit shape and appearance of hands. HandAvatar [11] developed a high-resolution variant of MANO to fit personalized hand shapes and further disentangled the implicit representations for hands into geometry, albedo, and illumination. HandNeRF [10] designed a pose-driven deformation field with pose-disentangled NeRF to reconstruct single or interacting hands from multi-view images. LiveHand [12] proposed a low-resolution rendering of NeRF together with a super-resolution module to achieve real-time performance.

Training Image

Mesh

Reconstruction

Hand Mesh

Identity

Encoder

Interaction

Detection

Cross-subject

Identity Maps

Neural Texture Maps

Feature

Scattering

Identity

Embeddings

Pose

Embeddings

Texture

Conditions

Texture

Decoder

Interacting Points

Positional

Encoder

Positional Embeddings

Geometry

Decoder

Features

Feature

Sampling

Interaction-Aware

Attention

Refinement

Module

& Splatting

Rendered Image

Interaction-Aware Gaussian Splatting Network

Texture Flow

Point-based 3D Representation and Rendering . There has been a growing interest in point-based neural rendering due to recent advancements in 3D Gaussian splatting (3DGS) [32], which has led to high-quality and real-time rendering speed. Since 3DGS is primarily designed for static scenes, many efforts [33, 34, 35, 36] are dedicated to expanding its applicability to free pose animation and rendering. 3D-PSHR [34] achieved real-time and photo-realistic hand reconstruction from large-scale multi-view videos based on 3D points splatting. Our method differs from 3D-PSHR as it allows instant single-view one-shot hand avatar reconstruction and saves the expensive computational cost of per-scene optimization.

3 Methodology

In this section, we introduce the details of the proposed two-stage framework for creating interacting hand avatars from a single image. The key ideas of our framework contain three aspects: (i) We address the lack of information caused by limited inputs by learning disentangled priors for hand poses, shapes, and textures (Sec. 3.1). (ii) We construct an interaction-aware Gaussian splatting network to handle both intra- and inter-hand interactions (Sec. 3.2). (iii) Leveraging invertible identity and neural texture maps, we reduce the time consumption of one-shot avatar reconstruction while simultaneously improving the quality of synthesized images (Sec. 3.3).

3.1 Disentangled 3D Hand Representation

To demonstrate the motivations of the proposed disentangled 3D hand representation, we first provide the formulation of our task.

Task formulation . Given a reference image of interacting hands I r , our task is to reconstruct an animatable two-hand avatar that can generate images of the hands with novel poses and from novel views. To achieve this, we propose to construct a differentiable renderer R : R | θ |×| c | → R H × W × 3 , where θ and c denote the hand pose parameters and camera parameters, respectively. H and W denote the height and width of rendered three-channel RGB images.

More specifically, we propose to implement R via Gaussian splatting (GS) [32] for its advantages in explicit modeling and computational efficiency. Essentially, GS is a point-based rasterization technique that leverages a set of 3D points (Gaussians) with attributes like colors, opacity, and spherical harmonic coefficients to represent reference images. Let P ∈ R N × 3 denote the set of N Gaussians and a ∈ R N × D denote their D-dimensional attributes. To preserve the characteristic of the hands in I r , we need to optimize R via minimizing the following l 1 loss,

Due to the high-dimensional attribute space and inter-attribute interference, direct optimization for Eq. (1) is intractable. Although learning-based methods can alleviate this issue to a certain extent, their

results may fall short if I r lies outside of the distribution (OOD) of their training data. Therefore, a disentangled representation that integrates optimization-based and learning-based Gaussian attributes is essential for addressing our task effectively. Furthermore, a reliable representation should capture both the geometric and textural properties of hands. Given the availability of extensive hand mesh reconstruction methods for geometric information, we propose leveraging learning-based geometric properties alongside optimizable textures in our disentangled representation. This approach allows us to handle the diversity and potential OOD issues of hand textures.

To this end, we devise the disentangled representation shown in Figure 3 to combine explicit geometric embeddings from hand meshes with neural texture maps encoding implicit latent fields. The encoders for this representation are learned on a training dataset consisting of images of S subjects. Specifically, we first construct the optimizable cross-subject identity maps m ∈ R S × 2 C × H × W , where C denotes the number of feature channels of one hand. For each training image I t , we reconstruct a parameterized hand mesh M from it and use the vertices of M to initialize P . Let s ∈ [1 , .., S ] denote the subject ID of I t , we retrieve its corresponding identity map m s and combine it with the pose embedding from M to infer its neural texture map t s ∈ R 2 C × H × W . Consequently, ∀ p ∈ P , we can query their feature vectors from t s based on their texture coordinates and predict their Gaussian attributes. In the one-shot fitting stage, to obtain the neural texture map of I r , we only need to optimize a new identity map initialized with zeros. This is applicable, as our identity maps and natural texture maps share an important advantage: they both preserve the spatial structure of textures, which overcomes the burdens of previous vector-based inversion methods [18, 37].

Below we introduce the core components for implementing our disentangled representation. These components will be integrated into the GS network in the next section for end-to-end learning.

Parameterized Hand Mesh . We employ MANO [1] to reconstruct hand meshes from images for its convenience in animation. MANO is a parametric model that represents a hand mesh by pose parameters θ ∈ R 48 and shape parameters β ∈ R 10 . To further improve the mesh quality, we use a high-resolution version of MANO [11]. We follow [12] to obtain normalized UV coordinates ( u, v ) ∈ [0 , 1] × [0 , 1] of mesh vertices and project them onto the neural texture plane.

Geometric Encoding . To exploit explicit geometric features from hand meshes, we utilize a pose encoder and a positional encoder. The pose encoder is an MLP taking the pose parameters θ and the camera parameters c ∈ R 25 as inputs (which is the flattened concatenation of an extrinsic matrix ∈ R 4 × 4 and an intrinsic matrix ∈ R 3 × 3 ). Our pose embeddings do not involve β and hence they are independent of identities. The positional encoder is a shallow PointNet [38] with local pooling [39]. We further employ a transformer-based decoder to merge the outputs of the pose encoder and the positional encoder, similar to [40].

Texture Encoding . Given the UV coordinates of a vertex, we retrieve its feature vector on the optimizable identity map, along with the γ positional encoding [26] of its coordinates to generate its identity embedding. The identity embeddings of all vertices are concatenated with the pose embedding and projected (scattered) back to the UV plane to form a texture condition map. We again adopt a transformer-based decoder to process the condition map and yield the neural texture map.

Finally, we combine the geometric feature vectors of all vertices with their texture feature vectors using element-wise addition. This results in a unified latent representation f ∈ R |P|× C , which we use for predicting Gaussian attributes.

3.2 Interaction-Aware Gaussian Splatting Network

To better reconstruct interacting hand avatars with various poses, we propose to enhance the Gaussian features f via an interaction-aware attention (IAttn) module and a Gaussian point refinement module (GRM). IAttn identifies potential points with intra- or inter-hand interaction. By exploring the context around interaction points, IAttn improves the reconstruction quality of geometric deformations and texture details resulting from interactions (such as shading, wrinkles, and veins). Furthermore, GRM not only eliminates redundant Gaussians but also generates additional Gaussians near regions with complex textures. With these two modules, our network can render high-quality hand images with rich details.

3.2.1 Interaction-aware Attention

To detect interacting points in P , we propose a straightforward yet effective strategy that calculates the difference between the neighboring point sets of posed hand meshes and a canonical mesh. This strategy is practical as we can define an interaction-free mesh as the canonical one. For an arbitrary query point q ∈ P , if its topN c nearest neighboring points on the canonical mesh Ω c ( q ) are significantly overlapped with its topN p nearest neighbors on the posed mesh (denoted as Ω p ), the chance that q is an interacting point is low. This strategy can be formulated as follows,

where T is a user-defined threshold. Additionally, we append the interacting label d ( q ) to the pose embedding introduced in the last section. Note that our proposed strategy can detect both selfinteracting and cross-interacting points. To maintain the efficiency of our method, we only conduct self-attention on detected interacting points.

3.2.2 Self-adaptive Refinement for 3D Gaussians

Gaussians initialized from MANO can only provide coarse hand geometry with restricted deformations. To better model the geometry of hands with various poses and shapes, we devise the self-adaptive GRM to control the density of Gaussians and refine their locations. Given a Gaussian point p and its corresponding feature vector f p , we utilize an MLP ϕ ( f p ) with the sigmoid activation to predict the validity of p , i.e., ϕ : R C → [0 , 1] . We remove p from P if ϕ ( f p ) is below a pre-defined threshold T d while splitting p if ϕ ( f p ) is larger than another threshold T s . We also exploit f to predict the offsets of Gaussians to adjust their positions.

3.2.3 Network Optimization

With the search space of Gaussian attributes reduced significantly by the proposed disentangled representation, we are now ready to optimize our GS network and learn cross-subject hand priors. We train the GS network along with the optimizable cross-subject identity maps m by minimizing the following loss:

where Θ are the learnable parameters in our GS network. L V GG denote the perceptual loss [41]. λ rgb and λ V GG are user-defined weights.

3.3 One-shot Hand Avatar Reconstruction

In the stage of one-shot hand avatar reconstruction, the parameters of IGSN are fixed and we fine-tune the identity map of the new subject m ∗ ∈ R 2 C × H × W . This can be formulated as,

where the geometric parameters θ r , c r , and P r can be obtained via off-the-shelf MANO regressors [42, 43]. M and M r denote the hand mask of I and I r , respectively. λ mask is the empirical weight of the loss term on masks.

[18] suggests that optimization tricks like color calibration and view regularization can further improve the synthesized images. Inspired by this, we also introduce a texture map bias ∆ t ∈ R C × H × W to modulate the latent neural feature maps t of two hands ( t = { t l , t r } ). That is, t l := t l +∆ t and t r := t r +∆ t . We assume that ∆ t can be shared by t l and t r due to the symmetry of the left and right hands. We include a regularization term on ∆ t into Eq. (4) to prevent drastic shifts of t as follows:

where the regularization weight λ reg is user-defined. In our experiments, we find that adding ∆ t accelerates the fitting process and helps to prevent undue changes.

4 Experiments

4.1 Setup

Learning IGSN. Our experiments are conducted on the publicly available Interhand2.6M dataset [19] (CC-BY-NC 4.0 licensed) that consists of large-scale multi-view sequences of different subjects performing various hand poses. Following [18], we adopt interacting-hands pose sequences of 21 subjects from InterHand2.6M training set for pre-training. For each subject, an unseen pose sequence is used for evaluation. Our network is trained on three A6000 GPUs using the Adam optimizer [44] with the learning rates of 1 × 10 -4 for eight epochs. Loss weights in Eq. (3) are set as λ rgb = 10 . 0 , λ V GG = 0 . 1 . For interaction detection, we set N c = 100 and T = 90 . For self-adaptive GRM, we set T d = 0 . 1 and T s = 0 . 9 . We adopt a coarse-to-fine mesh refinement strategy during training: For the first 5 epochs, we upsample hand meshes to 12,337 points per hand while for the last three epochs, we further upsample hand meshes to 49,281 points per hand.

One-shot Reconstruction. We conduct one-shot reconstruction evaluations on the testing set of InterHand2.6M as in [18, 11]. To evaluate the novel pose rendering quality, We evenly sample 349 frames from four pose sequences including four common views in the 'test/capture0' subset as the test set. To assess the quality of novel view synthesis, we have selected the initial 50 views from the "test/capture0" subset. The one-shot fitting takes 50 optimization steps with the learning rate of 1 × 10 -2 . The whole process takes 2.5 minutes with an A6000 GPU. Loss weights in Eq. (4,5) are set as λ mask = 1 . 0 , λ reg = 0 . 01 .

Baselines and Metrics. We select four state-of-the-art methods for comparison. We adopt two generalizable NeRFs including KeypointNeRF [15] designed for human novel view synthesis and VANeRF [13] designed for single-view interacting-hand image novel view synthesis. We further adapt VANeRF to the one-shot animatable interacting hands reconstruction. Moreover, We include SMPLpix [8] as an image-space baseline. Besides, although OHTA [18] is not designed for interacting hands reconstruction, we still implement its one-shot strategy with our pre-trained model (denoted as OHTA*) for one-shot performance comparison. Following previous works [13, 8, 15, 18], we report LPIPS[45], PSNR[46], and SSIM[47] as the metrics of rendering quality.

4.2 Comparison with State-of-the-art Methods

Quantitative Comparison. Table 1 reports the quantitative results of our method against the baselines in the one-shot reconstruction scenario, including novel view synthesis and novel pose synthesis. We can see that our method significantly outperforms all methods on all metrics in both tasks. NeRFbased methods, KeypointNeRF and VANeRF fail to have good performance facing large view or pose variations due to the under-utilization of the hand priors. SMPLpix as an image-space method lacks generalization and 3D understanding when coping with single-view reconstruction. Compared with OHTA, our method captures more accurate characteristics of the target identity using the identity map and neural map bias.

Qualitative Comparison. Figure 4 demonstrates the visual comparison between our approach and the baselines. SMPLpix fails to produce a reasonable hand appearance with the limited information from a single image. VANeRF predicts the basic hand geometry while leaving high-frequency details like wrinkles and veins. OHTA recovers a reasonable hand geometry with most of the appearance close to the target identity while failing to capture fine-grained identity features. Compared with baselines, our method successfully recovers hand details (e.g. nails, wrinkles, and veins) of the

GT

SMPLpix

VANeRF

Ours

OHTA*

GT

SMPLpix

VANeRF

Ours

OHTA*

天

target identity for various views and poses. The qualitative comparisons demonstrate our robust performance for one-shot animatable hand avatars creation.

Ours-I

Ours-II

Ours-III

Ours-IV

Ours

OHTA

*

Ours

w/o IAttn

w/o GRM

w/o IMap

Ours

w/o IAttn

w/o GRM

w/o IMap

0.1

0

4.3 Ablation Study

We conduct ablations studies in each of the two stages of our framework. Particularly, we focus on the components of IGSN in the first stage while the loss terms for one-shot fitting in the second stage.

4.3.1 Interaction-aware Gaussian Splatting

The evaluation is conducted in the 'train/capture0' subset with 23 pose sequences for training and 1 for testing. The quantitative results are reported in Table 2 Stage-One. The results reflect that each design does bring performance gains and the best performance is obtained by the full model.

Effectiveness of GRM. To validate the effectiveness of our Gaussian points refinement module, we implement a variant by substituting refined Gaussian points with upsampled hand mesh points (denoted as w/o GRM. in Table 2). To further verify the effectiveness of our points modification based on the points validation prediction, we implement a variant by replacing the Gaussian points

↓

refinement module with simply predicting offset for each point (denoted as w/o GPM. in Figure 5). The results indicate that the points modification eliminates the redundant points and densifies the detailed area based on point features, producing better rendering results.

GT

Ours

Interactions

GT

Ours

Interactions

GT

Ours

Interactions

食台

Effectiveness of IAttn. Figure 6 demonstrates the predictions of the proposed interaction detection in IAttn. We can observe that our designed interaction detection effectively recognizes the interaction areas, including one-hand self-interaction and the interaction between two hands.

To validate the effectiveness of the interaction-aware attention module, we remove it from our full model (denoted as w/o IAttn. in Table 2 and Figure 5). The results show that the proposed module improves the reconstruction quality of geometry deformation and texture details caused by interaction based on effective interaction detection.

Effectiveness of Identity Map. We also evaluate the effectiveness of the identity map by substituting it with the identity code (vector) as in [18] (denoted as w/o IMap. in Table 2 and Figure 5). The results demonstrate that the Identity Map contributes to high-fidelity hand reconstruction by capturing the fine-grained texture of the target identity.

4.3.2 One-shot Fitting

We show the effectiveness of our strategies for one-shot reconstruction in Table 2 Stage-Two and Figure 5. When the identity map is replaced with identity code (IV), the performance drops indicating that our identity map can capture more accurate and fine-grained characteristics of the target hand. When there is no neural texture map bias (III), the details of the target hands are missing. Without color calibration (II), the results become worse due to the color bias. When omitting the identity inversion (I), a significant drop in LPIPS is observed as the target identity information is not captured. Overall, we validate the effectiveness of our design for one-shot reconstruction.

5 Conclusion

We tackle the challenging single-image interacting hand avatar reconstruction task via an interactionaware Gaussian splatting framework in this paper. Our framework disentangles 3D hand representations into learning-based features that can be extracted by the trained network and identity maps

that are one-shot optimized on the hands of new subjects. Additionally, our framework employs an interaction-aware attention module and a self-adaptive refinement module to detect and handle regions with intra- and inter-hand interactions. The proposed method outperforms cutting-edge methods on the Interhand2.6M dataset and creates high-quality avatars for various tasks successfully.

6 Acknowledgments

This work was supported in part by National Key R&D Program of China (2022YFA1004100), National Science Foundation of China Grant No. 62176035, No. 62372482, No. 62476293, and No. 61936002, National Science and Technology Major Project (2020AAA0109704), Guangdong Outstanding Youth Fund (Grant No. 2021B1515020061), Shenzhen Science and Technology Program (Grant No. GJHZ20220913142600001), Nansha Key RD Program under Grant No.2022ZD014.

References

Javier Romero, Dimitrios Tzionas, and Michael J. Black. Embodied hands: modeling and capturing hands and bodies together. ACM Transactions on Graphics , 36(6), nov 2017.

Neng Qian, Jiayi Wang, Franziska Mueller, Florian Bernard, Vladislav Golyanik, and Christian Theobalt. Html: A parametric hand texture model for 3d hand reconstruction and personalization. In Proceedings of the European Conference on Computer Vision , pages 54-71. Springer, 2020.

Korrawe Karunratanakul, Sergey Prokudin, Otmar Hilliges, and Siyu Tang. Harp: Personalized hand reconstruction from a monocular rgb video. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 12802-12813, 2023.

Yuwei Li, Longwen Zhang, Zesong Qiu, Yingwenqi Jiang, Nianyi Li, Yuexin Ma, Yuyao Zhang, Lan Xu, and Jingyi Yu. Nimble: a non-rigid hand model with bones and muscles. ACM Transactions on Graphics , 41(4):1-16, 2022.

Rolandos Alexandros Potamias, Stylianos Ploumpis, Stylianos Moschoglou, Vasileios Triantafyllou, and Stefanos Zafeiriou. Handy: Towards a high fidelity 3d hand shape and appearance model. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 4670-4680, 2023.

Yujin Chen, Zhigang Tu, Di Kang, Linchao Bao, Ying Zhang, Xuefei Zhe, Ruizhi Chen, and Junsong Yuan. Model-based 3d hand reconstruction via self-supervised learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 10451-10460, 2021.

Zheheng Jiang, Hossein Rahmani, Sue Black, and Bryan M Williams. A probabilistic attention model with occlusion-aware texture regression for 3d hand reconstruction from a single rgb image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 758-767, 2023.

Sergey Prokudin, Michael J Black, and Javier Romero. Smplpix: Neural avatars from 3d human models. In Proceedings of the IEEE Winter Conference on Applications of Computer Vision , pages 1810-1819, 2021.

Enric Corona, Tomas Hodan, Minh Vo, Francesc Moreno-Noguer, Chris Sweeney, Richard Newcombe, and Lingni Ma. Lisa: Learning implicit shape and appearance of hands. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 20533-20543, 2022.

Zhiyang Guo, Wengang Zhou, Min Wang, Li Li, and Houqiang Li. Handnerf: Neural radiance fields for animatable interacting hands. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 21078-21087, 2023.

Xingyu Chen, Baoyuan Wang, and Heung-Yeung Shum. Hand avatar: Free-pose hand animation and rendering from monocular video. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 8683-8693, 2023.

Akshay Mundra, Jiayi Wang, Marc Habermann, Christian Theobalt, Mohamed Elgharib, et al. Livehand: Real-time and photorealistic neural hand rendering. In Proceedings of the IEEE International Conference on Computer Vision , pages 18035-18045, 2023.

Xuan Huang, Hanhui Li, Zejun Yang, Zhisheng Wang, and Xiaodan Liang. 3d visibility-aware generalizable neural radiance fields for interacting hands. Proceedings of the AAAI Conference on Artificial Intelligence , 2024.

Youngjoong Kwon, Dahun Kim, Duygu Ceylan, and Henry Fuchs. Neural human performer: Learning generalizable radiance fields for human performance rendering. Advances in Neural Information Processing Systems , 34:24741-24752, 2021.

Marko Mihajlovic, Aayush Bansal, Michael Zollhoefer, Siyu Tang, and Shunsuke Saito. Keypointnerf: Generalizing image-based volumetric avatars using relative spatial encoding of keypoints. In Proceedings of the European Conference on Computer Vision , pages 179-197, 2022.

Shoukang Hu, Fangzhou Hong, Liang Pan, Haiyi Mei, Lei Yang, and Ziwei Liu. Sherf: Generalizable human nerf from a single image. In Proceedings of the IEEE International Conference on Computer Vision , pages 9352-9364, 2023.

Kyungmin Jo, Gyumin Shim, Sanghun Jung, Soyoung Yang, and Jaegul Choo. Cg-nerf: Conditional generative neural radiance fields for 3d-aware image synthesis. In Proceedings of the IEEE Winter Conference on Applications of Computer Vision , pages 724-733, 2023.

Xiaozheng Zheng, Chao Wen, Zhuo Su, Zeran Xu, Zhaohu Li, Yang Zhao, and Zhou Xue. Ohta: One-shot hand avatar via data-driven implicit priors. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2024.

Gyeongsik Moon, Shoou-I Yu, He Wen, Takaaki Shiratori, and Kyoung Mu Lee. Interhand2.6m: A dataset and baseline for 3d interacting hand pose estimation from a single rgb image. In Proceedings of the European Conference on Computer Vision , pages 548-564, 2020.

Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. Smpl: A skinned multi-person linear model. ACM Transactions on Graphics , 34(6):1-16, 2015.

Thiemo Alldieck, Marcus Magnor, Bharat Lal Bhatnagar, Christian Theobalt, and Gerard Pons-Moll. Learning to reconstruct people in clothing from a single rgb camera. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 1175-1186, 2019.

Verica Lazova, Eldar Insafutdinov, and Gerard Pons-Moll. 360-degree textures of people in clothing from a single image. In 2019 International Conference on 3D Vision (3DV) , pages 643-653. IEEE, 2019.

Xiangyu Xu and Chen Change Loy. 3d human texture estimation from a single image with transformers. In Proceedings of the IEEE International Conference on Computer Vision , pages 13849-13858, 2021.

Badour AlBahar, Shunsuke Saito, Hung-Yu Tseng, Changil Kim, Johannes Kopf, and Jia-Bin Huang. Single-image 3d human digitization with shape-guided diffusion. In SIGGRAPH Asia 2023 Conference Papers , pages 1-11, 2023.

David Svitov, Dmitrii Gudkov, Renat Bashirov, and Victor Lempitsky. Dinar: Diffusion inpainting of neural textures for one-shot human avatars. In Proceedings of the IEEE International Conference on Computer Vision , pages 7062-7072, 2023.

Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM , 65(1):99-106, 2021.

Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelnerf: Neural radiance fields from one or few images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 4578-4587, 2021.

Jianchuan Chen, Wentao Yi, Liqian Ma, Xu Jia, and Huchuan Lu. Gm-nerf: Learning generalizable model-based neural radiance fields from multi-view images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2023.

Fangzhou Hong, Zhaoxi Chen, Yushi LAN, Liang Pan, and Ziwei Liu. EVA3d: Compositional 3d human generation from 2d image collections. In International Conference on Learning Representations , 2023.

Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul P Srinivasan, Howard Zhou, Jonathan T Barron, Ricardo Martin-Brualla, Noah Snavely, and Thomas Funkhouser. Ibrnet: Learning multi-view image-based rendering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 4690-4699, 2021.

Haotong Lin, Sida Peng, Zhen Xu, Yunzhi Yan, Qing Shuai, Hujun Bao, and Xiaowei Zhou. Efficient neural radiance fields for interactive free-viewpoint video. In SIGGRAPH Asia Conference Proceedings , 2022.

Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics , 42(4):1-14, 2023.

Wojciech Zielonka, Timur Bagautdinov, Shunsuke Saito, Michael Zollhöfer, Justus Thies, and Javier Romero. Drivable 3d gaussian avatars. arXiv preprint arXiv:2311.08581 , 2023.

Zheheng Jiang, Hossein Rahmani, Sue Black, and Bryan M Williams. 3d points splatting for real-time dynamic hand reconstruction. arXiv preprint arXiv:2312.13770 , 2023.

Zhiyin Qian, Shaofei Wang, Marko Mihajlovic, Andreas Geiger, and Siyu Tang. 3dgs-avatar: Animatable avatars via deformable 3d gaussian splatting. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2024.

Shoukang Hu and Ziwei Liu. Gauhuman: Articulated gaussian splatting from monocular human videos. arXiv preprint arXiv:2312.02973 , 2023.

Marcel C Bühler, Kripasindhu Sarkar, Tanmay Shah, Gengyan Li, Daoye Wang, Leonhard Helminger, Sergio Orts-Escolano, Dmitry Lagun, Otmar Hilliges, Thabo Beeler, et al. Preface: A data-driven volumetric prior for few-shot ultra high-resolution face synthesis. In Proceedings of the IEEE International Conference on Computer Vision , pages 3402-3413, 2023.

Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 652-660, 2017.

Songyou Peng, Michael Niemeyer, Lars Mescheder, Marc Pollefeys, and Andreas Geiger. Convolutional occupancy networks. In Proceedings of the European Conference on Computer Vision , pages 523-540. Springer, 2020.

Zi-Xin Zou, Zhipeng Yu, Yuan-Chen Guo, Yangguang Li, Ding Liang, Yan-Pei Cao, and Song-Hai Zhang. Triplane meets gaussian splatting: Fast and generalizable single-view 3d reconstruction with transformers. arXiv preprint arXiv:2312.09147 , 2023.

Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In Proceedings of the European Conference on Computer Vision , pages 694-711, 2016.

Mengcheng Li, Liang An, Hongwen Zhang, Lianpeng Wu, Feng Chen, Tao Yu, and Yebin Liu. Interacting attention graph for single image two-hand reconstruction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 2761-2770, 2022.

Zhengdi Yu, Shaoli Huang, Chen Fang, Toby P Breckon, and Jue Wang. Acr: Attention collaboration-based regressor for arbitrary two-hand reconstruction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 12955-12964, 2023.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 586-595, 2018.

Umme Sara, Morium Akter, and Mohammad Shorif Uddin. Image quality assessment through fsim, ssim, mse and psnr-a comparative study. Journal of Computer and Communications , 7(3):8-18, 2019.

Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing , 13(4):600-612, 2004.

Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 3836-3847, 2023.

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. arXiv:2304.02643 , 2023.

A Appendix

A.1 Application.

As shown in Figure 7, We demonstrate more visual results of various applications, including in-the-wild results from real-captured images, text-to-avatar, and texture editing. For text-to-avatar, we utilize ControlNet [48] with depth maps as condition information for hand image generation. For in-the-wild results, we use ACR [43] to estimate hand pose and camera parameters from real-captured images. These results clearly show that our method can be applied to in-the-wild images and obtain considerable results.

'hand,

spider man'

'hand,

blue'

'hand,

old'

'hand,

pale'

Text-to-avatar

In-the-wild

Editing

北业

A.2 Statistical Significance

To evaluate the statistical significance, we run the proposed method five times with different random initiation of identity map in the second stage. Table 3 presents the mean and standard deviation of the results.

A.3 Visualization of GRM

Figure 8 presents the qualitative comparisons of hand geometry and rendering results between our Gaussian refinement modules and vanilla GS with mesh upsampling. We can see that, the hand points initialized from hand parameters can only provide approximate hand geometry while refined Gaussian points excel in producing more detailed texture and realistic hand geometry with the help of the Gaussian points Refinement module.

MeshUp | GRM

MeshUp

GRM

GT

MeshUp | GRM

MeshUp

GRM

GT

A.4 Data Selection and Preprocessing

We use the same captures for training with OHTA [18] in stage one and save the pose sequence '0275_left-babybird' for testing. For the evaluation of stage two, we conduct experiments on 'test/Capture0' including four pose sequences 'ROM01_No_Interaction_2_Hand', 'ROM01_No_Interaction_2_Hand', 'ROM09_Interaction_Fingers_Touching', and 'ROM09_Interaction_Fingers_Touching_2'.

For image preprocessing, we crop out the hand region with the bounding boxes and resize the cropped area to 256 × 256 consistent with previous methods [13, 18]. We adopt SAM [49] to produce hand masks for better segmentation compared to MANO-rendered masks.

A.5 More Ablations

Figure 9 demonstrates visual examples of shadow disentanglement (top) and qualitative results of four ablation studies (bottom). The quantitative results of the four ablation studies are provided in Table 4.

Shadow

Shaded Image

Shadow

Albedo

Shaded Image

Shadow

Albedo

Shaded Image

Shadow

Albedo

Shaded Image

Shadow

Albedo

Reference

Reference

Reference

Rendering

Rendering

Rendering

Reference

Reference

Rendering

Rendering

Rendering

Rendering

Ours

Reference

w/ Noisy Mesh

GT

w/ Shadow

Ours

Ablation Hand Num.

Ablation Mask

Ablation Mesh

Ours

w/ Noisy Mesh

美

Shadow Disentanglement (denoted as Shadow). Figure 9 (top) demonstrates visual examples of shadow disentanglement including a shaded image, an albedo image, and a shadow image for each pair of hands. This figure clearly shows the shadow (dark) areas, which suggests that our method can disentangle albedo and shadow.

Ablation study on single-hand images (denoted as Ablation Hand Num.). The advantage of using images of both hands is that two-hand images can provide more complementary information compared with single-hand images, which helps improve the model's performance. We experiment to compare the performance between single-hand images and two-hand images by masking one of the hands from the reference image. The results show that our method achieves better performance compared with the single-hand baseline, which is reasonable as two-hand images contain more complementary information.

Ablation study on segmentation method (denoted as Ablation Mask). To validate the impact of segmentation masks, we consider masks predicted by SAM and the ground-truth meshes. We observe that SAM masks enhance the performance of our model, since they are better aligned with hands and prevent background clutters.

Ablation S1. Ablation S1 shows the visual results of three different ablation settings: including (i) the use of camera parameters, which is a variant of our method without the camera parameters (w/o Cam.) and suffers from obvious performance drop; (ii) adding shadow coefficient, which incorporates shadow coefficient prediction into the proposed method (w/ Shadow); (iii) Low Gaussian Points, which reduces the number of Gaussian points to 24k to show that coarsen hand boundaries are caused by fewer numbers of Gaussian points.

Ablation study on mesh estimation method (denoted as Ablation Noise). We use hand parameters estimated by ACR [43] to analyze how the accuracy of mesh reconstruction impacts the final results. The performance of our method with the predicted parameters is still satisfying, which suggests our method is robust to noise hand meshes to a certain extent.

A.6 Implementation of OHTA*

Since OHTA [18] as a single-hand reconstruction method cannot be directly applied to our scenario, we instead use our pre-trained model and only compare the design of the fine-tuning stage. We implement OHTA* with the texture inversion stage and texture fitting stage as described in its paper. In the texture inversion, we keep the network weights frozen and optimize the identity code along with per-channel color calibration coefficients to produce a similar appearance to the target identity of the input image. In the texture fitting, we fine-tune the texture feature MLP for texture feature extraction and constrain the texture-fitting results of some reference views to be close to the rendering results before texture-fitting.

A.7 Limitations and Society Impact

Although we have greatly shortened the fine-tuning process, the per-identity training still limits the application compared to single forward inference. Moreover, separate optimization for each identity hinders the model from integrating similar identity information optimized in the fine-tuning stage.

When facing serious errors in pose estimation, our method may fail to model the proper appearance through the wrong alignment caused by misleading hand parameters as shown in Figure 10. Severe estimation errors inevitably cause degraded performance, which is also mentioned in the paper of OHTA [18].

Our method has a positive impact on society as it can facilitate sign language production by creating high-fidelity, animatable hand avatars with interaction. There are no negative societal impacts concerning our work.

Rendering

Reference

Keypoints

Mask

Masked image

Mesh

Rendering

Reference

Keypoints

Mask

Masked image

Mesh

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The paper's contributions and scope are accurately stated in the abstract and at the end of the introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The discussion of limitations is involved in the appendix (Sec. 6).

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include theoretical results or formal proofs.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The setup of experiments is provided in Setup (Sec. 4.1) for reproduction.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Code and models will be released upon acceptance.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/public/ guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https://nips.cc/public/ guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The training and test details are provided in Setup (Sec. 4.1).

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The error bars are reported in the appendix (Sec. 6).

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The information on computer resources is provided in Setup (Sec. 4.1).

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The paper discuss both potential positive societal impacts and negative societal impacts of the work performed in the appendix (Sec. 6).

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The creators or original owners of assets used in the paper are properly credited and the license and terms of use are explicitly mentioned in the Setup (Sec. 4.1).

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a method to create a 3D Gaussian Splatting (GS)-based avatar for interacting hands from single-image inputs. Its main contribution is to propose a two-stage GS framework (1) to leverage cross-identity priors via learning-based features and also to (2) well preserve per-identity information via optimized identity maps. Additionally, it proposes an interaction-aware attention module and a Gaussian refinement module to further exploit the hand interaction context for more realistic rendering. In the experiments, the proposed methods outperforms the existing methods based on generalizable and one-shot NeRFs.\n\nSTRENGTHS:\n**(1) Good presentation.** Overall, the paper is well-organized and technical explanations are sound. The figures are also clearly presented.\n\n**(2) Sufficient technical novelty.** The disentanglement of hand representation into learning-based features and optimization-based identity maps sounds reasonable. Also, to the best of my knowledge, this is one of the first works on generating avatars for interacting hands from single image inputs.\n\n**(3) Good experimental validation.** The paper reports sufficient ablation study results and performs adequate comparisons with the existing baselines.\n\nWEAKNESSES:\n**(1) Effects of off-the-shelf MANO regressor and mask detector.** The proposed avatar reconstruction method (Sec. 3.3) uses the outputs of off-the-shelf MANO regressor and segmentation model. It would be informative to include the discussions on how the quality of these outputs might affect the final rendering quality.\n\n**(2) Non-smooth hand boundaries in rendering results.** In Fig. 4, I’ve noticed that the boundaries of hands rendered using the proposed method are less smooth compared to the existing methods. Does it indicate that the proposed GS-based method (implicitly) learns less smooth hand geometries than the other NeRF or mesh-based methods? More discussions regarding this would be informative.\n\nQUESTIONS:\n(1) In lines 94-95, the paper claims that the rendering quality of the existing MANO-based methods “is constrained by the coarse geometry and sparsity of the MANO mesh”. However, to my knowledge, HARP [3] and Handy [5] use higher resolution-version of MANO meshes to already address this issue. What would be the advantage of using GS compared to using such higher-resolution hand meshes?\n\n(2) In lines 118-119, the paper states that the proposed method \"reduce[s] the time consumption of one-shot avatar reconstruction\". Have you tried comparing the reconstruction time of the proposed method against OHTA [18]?\n\nLIMITATIONS:\nThe authors discussed the limitations in Sec. A.4 in Appendix.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes an approach to achieve one-shot interacting hand avatar reconstruction via 3D Gaussian Splatting. The authors design a two-stage framework: the first stage learns learning-based features and optimization-based identity maps, and the second stage performs one-shot reconstruction by optimizing only the identity map. To better capture the relationship between the two hands, they devise an interaction-aware attention module. Moreover, they design a self-adaptive GS density control approach for this task. The experiments demonstrate the effectiveness of their designs and show that the approach achieves state-of-the-art performance.\n\nSTRENGTHS:\n1. The paper presents the first framework capable of achieving one-shot animatable two-hand avatars.\n2. The disentangled designs for one-shot reconstruction, using optimization-based identity maps for the one-shot stage, are technically sound.\n\nWEAKNESSES:\n1. For interacting hands, significant shadows occur between the two hands. The proposed method does not account for shadow modeling, which overlooks the dynamic nature of the hands and makes the approach less practical for real-world scenarios to some extent.\n2. The experimental results are not comprehensive.\n- It would be beneficial to present some in-the-wild testing results to demonstrate the method's plausibility for real applications. This would also justify the design for addressing out-of-distribution (OOD) data.\n- Including some failure cases would help to illustrate the boundaries of the work.\n\nQUESTIONS:\n1. Why should we use interacting hands for avatar creation? What are the benefits compared to single-hand modeling?\n2. What is the detailed implementation of OHTA? The results appear to be worse than the reported results.\n3. How does the accuracy of mesh reconstruction impact the final results?\n\nLIMITATIONS:\nThe authors present the \"Limitations and Society Impact\" section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel two-stage interaction-aware GS framework to create animatable avatars for interacting hands from single-image inputs. The proposed method disentangle the 3D presentation of hands into optimization-based identity maps and learning-based latent geometric features and neural texture maps, and exploits cross-subject hand priors and refines 3D Gaussians in interacting areas. Experiments are conducted on the Interhand2.6M dataset with state-of-the-art method comparisons.\n\nSTRENGTHS:\n1) The paper is well written and easy to follow, and it is technically sound to disentangle geometric and texture feature learning to achieve prior learning across different subjects.\n2) The idea of the Interaction-aware Attention module is novel, which detect the interation-aware geometric deformation in an intuitive way.\n3) The quantitative and qualitative results are of good quality and surpass the current SOTA on Interhand2.6M, and the ablation studies are quite extensive.\n4) Comprehensive qualitative ablation studies are shown. The authors additionally present qualitative results on the interaction areas, showing enhanced rendering quality.\n\nWEAKNESSES:\n1) The data flow in Figure 3 is somewhat unclear. For example, what does the dotted line from the Interaction Points to the Interaction-Aware Attention present? In addition, using different colored sub boxes for geometric flow and texture flow can help readers more intuitively understand the disentanglement of hand priors.\n2) In line 119 of page 4, there is a typo error. 'Sec.' is missing.\n3) The generalization of in-the-wild images has not been demonstrated.\n\nQUESTIONS:\n1) In lines 85-87 of page 3, why can disentangling hand prior learning reduce the cost of one-shot fitting？Is this related to introducing texture map bias to accelerate one shot fitting\n2) Why is the interacting label $d(q)$ binarized, rather than applied by soft activation?\n3) An important baseline, OHTA, was re-implemented and compared in the experiment. Can you provide more details?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n- The author extends the concept of one-shot hand avatar creation from the single hand in the previous OHTA paper to two hands.\n- The author proposes a novel two-stageGS framework for the reconstruction and rendering of avatars. This framework utilizes 2D identity maps to represent identity information and assist in the learning of neural textures. Additionally, an attention mechanism and a Gaussian refinement module are designed to handle the interaction information.\n- The author demonstrates the superior performance of their method compared to the baseline on the Interhand2.6M dataset.\n\nSTRENGTHS:\n- The author has successfully applied the Gaussian splatting module for the first time in the problem of one-shot hand avatar creation.\n\n- The method proposed in the paper has shown improved performance on the Interhand dataset.\n\n- The paper is easy to follow.\n\nWEAKNESSES:\n- The article represents an incremental work.\n\n- Initially, it is not convincing why one would construct a hand avatar using images of both hands with interaction instead of building them separately. What are the advantages?\n\n- Secondly, the advantage of the one-shot method lies in addressing in-the-wild scenarios. The pioneering work in the single hand direction, OHTA, demonstrated a multitude of in-the-wild results, but this paper does not. This raises doubts about the effectiveness of this method in in-the-wild scenarios.\n\n- The method's modeling of the hand is not sufficiently detailed. Previous methods such as Handavatar, OHTA, etc., have considered shadows caused by occlusions, but it is not clear how this article's method learns such priors.\n\nQUESTIONS:\n- The OHTA* method compared in the article seems more like an ablation of the full model. The performance of the OHTA algorithm reproduced by the authors is worse than in the original paper. Can an explanation be provided on how this baseline was implemented, and can some reasons for the poorer performance be analyzed?\n\n- The paper incorporates camera intrinsic and extrinsic parameters as part of the Geometric Encoding. What are the advantages of introducing this information? Is there any related ablation analysis?\n\n- Is the initial mesh of the bimanual results shown by the author estimated or taken from the dataset's ground truth? If it is estimated, what estimation method was used? How is the error in pose estimation addressed?\n\nLIMITATIONS:\nThe author has already addressed the limitations of the method.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}