Learning Interaction-aware 3D Gaussian Splatting for One-shot Hand Avatars

Authors: xuan  huang
hanhui  li
wanquan  liu
xiaodan  liang
yiqiang  yan
yuhao  cheng
chengqiang  gao
  mesh
positional  encoder
positional  embeddings
gaussian  decoder
  splatting
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 06:56:06

Abstract: In this paper, we propose to create animatable avatars for interacting hands with 3D Gaussian Splatting (GS) and single-image inputs. Existing GS-based methods designed for single subjects often yield unsatisfactory results due to limited input views, various hand poses, and occlusions. To address these challenges, we introduce a novel two-stage interaction-aware GS framework that exploits cross-subject hand priors and refines 3D Gaussians in interacting areas. Particularly, to handle hand variations, we disentangle the 3D presentation of hands into optimization-based identity maps and learning-based latent geometric features and neural texture maps. Learning-based features are captured by trained networks to provide reliable priors for poses, shapes, and textures, while optimization-based identity maps enable efficient one-shot fitting of out-of-distribution hands. Furthermore, we devise an interaction-aware attention module and a self-adaptive Gaussian refinement module. These modules enhance image rendering quality in areas with intra- and inter-hand interactions, overcoming the limitations of existing GS-based methods. Our proposed method is validated via extensive experiments on the large-scale InterHand2.6M dataset, and it significantly improves the state-of-the-art performance in image quality. Code and models will be released upon acceptance.

Main paper: 
 introduction: 
 Recent advancements in 3D reconstruction and differential rendering techniques have significantly improved hand avatar creation and related applications. However, creating avatars for interacting hands from a single image remains challenging. The limited input view does not provide sufficient geometry and texture information for accurate reconstruction. Moreover, intra-and inter-hand interactions exacerbate information loss and introduce complex geometric deformations.Extensive efforts have been made to tackle these issues, as shown in Figure 2: (a) Early approaches depend on explicit parametric meshes (e.g. MANO [1] ) for geometry modeling, and utilize UV map [2, 3, 4, 5] , vertex color [6, 7] , or image space rendering [8] for appearance. Despite the efficiency in rendering, these methods fail to achieve realistic rendering results with the coarse mesh resolution and the simple combination of hand appearance and geometry. (b) More recently, We present a novel interaction-aware Gaussian splatting framework that creates animatable interacting hand avatars from a single image. These high-fidelity avatars support various applications, such as editing, animation, combination, duplication, re-scaling, and text-to-avatar conversion.with the significant success of neural radiance fields (NeRF), extensive studies [9, 10, 11, 12, 13] have employed NeRF-based models for implicit modeling. These methods [10, 11, 12] usually require per-scene optimization for each new identity using densely calibrated images, which results in expensive training costs. Generalizable NeRFs [14, 15, 13, 16] get rid of per-scene training by leveraging image-aligned features to enable reconstruction from a few or even a single view. Yet their dependence on image-aligned features also limits their performance under large view or pose variations. Besides, (c) One-shot NeRF-based methods [17, 18] propose to exploit data-driven priors with condition optimizations [17] and inversions [18] . Nevertheless, these methods are not suitable for our task, as they do not include any module to detect and handle interactions. Moreover, the inversion of identity vectors used in [18] omit the spatial image structure, which not only hinders its performance but also introduces extra time consumption for fine-tuning networks.To tackle the above issues in existing methods, we aim to create animatable avatars for interacting hands with 3D Gaussian Splatting (GS) and single-image inputs. To this end, we introduce a novel two-stage interaction-aware GS framework as shown in Figure 2 (d) . We disentangle the 3D presentation of hands into (i) features that can be effectively captured by training networks in the first stage of our framework (e.g., geometric features and latent neural texture maps), and (ii) identity maps that can be optimized efficiently in the per-subject one-shot fitting stage. In this way, our method not only enables leveraging cross-subject priors with learning-based features, but also well preserves per-subject characteristics via optimizing identity maps.with attention mechanisms. This enhancement allows our GS network to better model geometric deformations and fine-grained textures caused by interactions (e.g., wrinkles). The latter module is introduced to address the limitations of the coarse geometry of parametric hand meshes, which is achieved by learning to eliminate redundant Gaussians and assigning extra Gaussians in regions with complex textures and deformations. Consequently, our method can reconstruct realistic inter-hand avatars with great flexibility for animation and editing, as shown in Figure 1 .Overall, our contributions can be summarized as follows:• We propose a novel two-stage interaction-aware GS framework to create animatable avatars for interacting hands from single-image inputs. Our method generates high-fidelity rendering results and supports various applications. Experimental results on the large-scale Interhand2.6M dataset [19] validate the superior performance of our method compared to previous methods.• We disentangle the 3D presentation of hands into learning-based features that can be generalized well to different subjects and identity maps that are individually optimized for each subject. This disentanglement provides us with flexible and reliable priors for poses, shapes, and textures.• We introduce an interaction-aware attention module, which identifies intra-and inter-hand interactions and further exploits interaction context to improve rendering quality.• We devise a Gaussian refinement module that adaptively adjusts the number and positions of 3D Gaussian, which results in rendered images of higher quality under various hand poses and shapes. 
related work: 
 One-shot Human Reconstruction. One-shot reconstruction of 3D humans is a challenging and longstanding problem due to the limited information from a single input image. To alleviate this limitation, previous works leveraged parametric models [20, 1] as coarse geometry prior. Traditional methods [21, 22, 23, 24, 25] utilized UV maps for human appearance representation. To complete the unseen texture from the single image, some approaches [25, 24] inpainted missing textures via pre-trained diffusion models. Neural Radiance Fields (NeRF) [26] have also been explored for reconstruction from sparse views [27, 15, 14, 28] or a single view [29, 16, 13, 18] . KeypointNeRF [15] encoded spatial information using 3D skeleton keypoints. SHERF [16] created 3D human avatars from a single image with hierarchical features for informative encoding. VANeRF [13] leveraged visibility in both feature fusion and adversarial learning for single-view interacting-hand image synthesis. These methods greatly enhance the NeRF one-shot reconstruction performance. Nonetheless, generalizable NeRF [27, 30, 31, 14, 15] fails to achieve satisfactory results when the input image information is not sufficient for novel view prediction due to the under-utilization of hand priors. To overcome this issue, the pioneering research in [18] enabled one-shot single-hand avatar creation by learning data-driven hand priors which are further utilized with inversion and fitting. Compared with previous methods, our approach further disentangles the presentation of hand priors into latent geometric features, neural texture maps, and optimizable identity maps to enhance rendering quality and reduce the cost of one-shot fitting. Moreover, we introduce the interaction-aware module and self-adaptive refinement module, which helps to significantly improve the visual quality of synthesized images.Animatable Hand Avatar. Conventional methods created hand avatars by incorporating UV textures with explicit parametric hand models. HTML [2] is the first parametric texture model of human hands which models hand appearance with several dimensions of variability. HARP [3] further introduced albedo and normal information into UV maps to represent hand appearance without any neural components. Handy [5] realistically captured high-frequency detailed texture using a GANbased texture model. However, the rendering quality of these methods is constrained by the coarse geometry and sparsity of the MANO mesh. The recent advancements in the neural radiance field have resulted in the development of approaches that utilize implicit representation for hand reconstruction. LISA [9] is the first method that employs NeRF to learn the implicit shape and appearance of hands. HandAvatar [11] developed a high-resolution variant of MANO to fit personalized hand shapes and further disentangled the implicit representations for hands into geometry, albedo, and illumination. HandNeRF [10] designed a pose-driven deformation field with pose-disentangled NeRF to reconstruct single or interacting hands from multi-view images. LiveHand [12] proposed a low-resolution rendering of NeRF together with a super-resolution module to achieve real-time performance.Figure 3 : The architecture of the proposed interaction-aware Gaussian splatting network, of which the core components are the disentangled hand representation, the interaction detection module, the interaction-aware attention module, and the Gaussian refinement module labeled in green.Point-based 3D Representation and Rendering. There has been a growing interest in point-based neural rendering due to recent advancements in 3D Gaussian splatting (3DGS) [32] , which has led to high-quality and real-time rendering speed. Since 3DGS is primarily designed for static scenes, many efforts [33, 34, 35, 36] are dedicated to expanding its applicability to free pose animation and rendering. 3D-PSHR [34] achieved real-time and photo-realistic hand reconstruction from large-scale multi-view videos based on 3D points splatting. Our method differs from 3D-PSHR as it allows instant single-view one-shot hand avatar reconstruction and saves the expensive computational cost of per-scene optimization.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: The discussion of limitations is involved in the appendix (Sec. 6).Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
methodology: 
 In this section, we introduce the details of the proposed two-stage framework for creating interacting hand avatars from a single image. The key ideas of our framework contain three aspects: (i) We address the lack of information caused by limited inputs by learning disentangled priors for hand poses, shapes, and textures (Sec. 3.1). (ii) We construct an interaction-aware Gaussian splatting network to handle both intra-and inter-hand interactions (Sec. 3.2). (iii) Leveraging invertible identity and neural texture maps, we reduce the time consumption of one-shot avatar reconstruction while simultaneously improving the quality of synthesized images (Sec. 3.3).To demonstrate the motivations of the proposed disentangled 3D hand representation, we first provide the formulation of our task.Task formulation. Given a reference image of interacting hands I r , our task is to reconstruct an animatable two-hand avatar that can generate images of the hands with novel poses and from novel views. To achieve this, we propose to construct a differentiable renderer R : R |θ|×|c| → R H×W ×3 , where θ and c denote the hand pose parameters and camera parameters, respectively. H and W denote the height and width of rendered three-channel RGB images.More specifically, we propose to implement R via Gaussian splatting (GS) [32] for its advantages in explicit modeling and computational efficiency. Essentially, GS is a point-based rasterization technique that leverages a set of 3D points (Gaussians) with attributes like colors, opacity, and spherical harmonic coefficients to represent reference images. Let P ∈ R N ×3 denote the set of N Gaussians and a ∈ R N ×D denote their D-dimensional attributes. To preserve the characteristic of the hands in I r , we need to optimize R via minimizing the following l 1 loss,EQUATIONDue to the high-dimensional attribute space and inter-attribute interference, direct optimization for Eq. ( 1) is intractable. Although learning-based methods can alleviate this issue to a certain extent, their results may fall short if I r lies outside of the distribution (OOD) of their training data. Therefore, a disentangled representation that integrates optimization-based and learning-based Gaussian attributes is essential for addressing our task effectively. Furthermore, a reliable representation should capture both the geometric and textural properties of hands. Given the availability of extensive hand mesh reconstruction methods for geometric information, we propose leveraging learning-based geometric properties alongside optimizable textures in our disentangled representation. This approach allows us to handle the diversity and potential OOD issues of hand textures.To this end, we devise the disentangled representation shown in Figure 3 to combine explicit geometric embeddings from hand meshes with neural texture maps encoding implicit latent fields. The encoders for this representation are learned on a training dataset consisting of images of S subjects. Specifically, we first construct the optimizable cross-subject identity maps m ∈ R S×2C×H×W , where C denotes the number of feature channels of one hand. For each training image I t , we reconstruct a parameterized hand mesh M from it and use the vertices of M to initialize P. Let s ∈ [1, .., S] denote the subject ID of I t , we retrieve its corresponding identity map m s and combine it with the pose embedding from M to infer its neural texture map t s ∈ R 2C×H×W . Consequently, ∀p ∈ P, we can query their feature vectors from t s based on their texture coordinates and predict their Gaussian attributes. In the one-shot fitting stage, to obtain the neural texture map of I r , we only need to optimize a new identity map initialized with zeros. This is applicable, as our identity maps and natural texture maps share an important advantage: they both preserve the spatial structure of textures, which overcomes the burdens of previous vector-based inversion methods [18, 37] .Below we introduce the core components for implementing our disentangled representation. These components will be integrated into the GS network in the next section for end-to-end learning.Parameterized Hand Mesh. We employ MANO [1] to reconstruct hand meshes from images for its convenience in animation. MANO is a parametric model that represents a hand mesh by pose parameters θ ∈ R 48 and shape parameters β ∈ R 10 . To further improve the mesh quality, we use a high-resolution version of MANO [11] . We follow [12] to obtain normalized UV coordinates(u, v) ∈ [0, 1] × [0, 1]of mesh vertices and project them onto the neural texture plane.Geometric Encoding. To exploit explicit geometric features from hand meshes, we utilize a pose encoder and a positional encoder. The pose encoder is an MLP taking the pose parameters θ and the camera parameters c ∈ R 25 as inputs (which is the flattened concatenation of an extrinsic matrix ∈ R 4×4 and an intrinsic matrix ∈ R 3×3 ). Our pose embeddings do not involve β and hence they are independent of identities. The positional encoder is a shallow PointNet [38] with local pooling [39] .We further employ a transformer-based decoder to merge the outputs of the pose encoder and the positional encoder, similar to [40] .Texture Encoding. Given the UV coordinates of a vertex, we retrieve its feature vector on the optimizable identity map, along with the γ positional encoding [26] of its coordinates to generate its identity embedding. The identity embeddings of all vertices are concatenated with the pose embedding and projected (scattered) back to the UV plane to form a texture condition map. We again adopt a transformer-based decoder to process the condition map and yield the neural texture map.Finally, we combine the geometric feature vectors of all vertices with their texture feature vectors using element-wise addition. This results in a unified latent representation f ∈ R |P|×C , which we use for predicting Gaussian attributes.To better reconstruct interacting hand avatars with various poses, we propose to enhance the Gaussian features f via an interaction-aware attention (IAttn) module and a Gaussian point refinement module (GRM). IAttn identifies potential points with intra-or inter-hand interaction. By exploring the context around interaction points, IAttn improves the reconstruction quality of geometric deformations and texture details resulting from interactions (such as shading, wrinkles, and veins). Furthermore, GRM not only eliminates redundant Gaussians but also generates additional Gaussians near regions with complex textures. With these two modules, our network can render high-quality hand images with rich details.To detect interacting points in P, we propose a straightforward yet effective strategy that calculates the difference between the neighboring point sets of posed hand meshes and a canonical mesh. This strategy is practical as we can define an interaction-free mesh as the canonical one. For an arbitrary query point q ∈ P, if its top-N c nearest neighboring points on the canonical mesh Ω c (q) are significantly overlapped with its top-N p nearest neighbors on the posed mesh (denoted as Ω p ), the chance that q is an interacting point is low. This strategy can be formulated as follows,EQUATIONwhere T is a user-defined threshold. Additionally, we append the interacting label d(q) to the pose embedding introduced in the last section. Note that our proposed strategy can detect both selfinteracting and cross-interacting points. To maintain the efficiency of our method, we only conduct self-attention on detected interacting points.Gaussians initialized from MANO can only provide coarse hand geometry with restricted deformations. To better model the geometry of hands with various poses and shapes, we devise the self-adaptive GRM to control the density of Gaussians and refine their locations. Given a Gaussian point p and its corresponding feature vector f p , we utilize an MLP ϕ(f p ) with the sigmoid activation to predict the validity of p, i.e., ϕ : R C → [0, 1]. We remove p from P if ϕ(f p ) is below a pre-defined threshold T d while splitting p if ϕ(f p ) is larger than another threshold T s . We also exploit f to predict the offsets of Gaussians to adjust their positions.With the search space of Gaussian attributes reduced significantly by the proposed disentangled representation, we are now ready to optimize our GS network and learn cross-subject hand priors. We train the GS network along with the optimizable cross-subject identity maps m by minimizing the following loss:EQUATIONwhere Θ are the learnable parameters in our GS network. L V GG denote the perceptual loss [41] . λ rgb and λ V GG are user-defined weights.In the stage of one-shot hand avatar reconstruction, the parameters of IGSN are fixed and we fine-tune the identity map of the new subject m * ∈ R 2C×H×W . This can be formulated as,EQUATIONwhere the geometric parameters θ r , c r , and P r can be obtained via off-the-shelf MANO regressors [42, 43] . M and M r denote the hand mask of I and I r , respectively. λ mask is the empirical weight of the loss term on masks.[18] suggests that optimization tricks like color calibration and view regularization can further improve the synthesized images. Inspired by this, we also introduce a texture map bias ∆t ∈ R C×H×W to modulate the latent neural feature maps t of two hands (t = {t l , t r }). That is, t l := t l + ∆t and t r := t r + ∆t. We assume that ∆t can be shared by t l and t r due to the symmetry of the left and right hands. We include a regularization term on ∆t into Eq. ( 4) to prevent drastic shifts of t as follows: arg min For each subject, an unseen pose sequence is used for evaluation. Our network is trained on three A6000 GPUs using the Adam optimizer [44] with the learning rates of 1 × 10 -4 for eight epochs. Loss weights in Eq. ( 3) are set as λ rgb = 10.0, λ V GG = 0.1. For interaction detection, we set N c = 100 and T = 90. For self-adaptive GRM, we set T d = 0.1 and T s = 0.9. We adopt a coarse-to-fine mesh refinement strategy during training: For the first 5 epochs, we upsample hand meshes to 12,337 points per hand while for the last three epochs, we further upsample hand meshes to 49,281 points per hand.One-shot Reconstruction. We conduct one-shot reconstruction evaluations on the testing set of InterHand2.6M as in [18, 11] . To evaluate the novel pose rendering quality, We evenly sample 349 frames from four pose sequences including four common views in the "test/capture0" subset as the test set. To assess the quality of novel view synthesis, we have selected the initial 50 views from the "test/capture0" subset. The one-shot fitting takes 50 optimization steps with the learning rate of 1 × 10 -2 . The whole process takes 2.5 minutes with an A6000 GPU. Loss weights in Eq. (4,5) are set as λ mask = 1.0, λ reg = 0.01.Baselines and Metrics. We select four state-of-the-art methods for comparison. We adopt two generalizable NeRFs including KeypointNeRF [15] designed for human novel view synthesis and VANeRF [13] designed for single-view interacting-hand image novel view synthesis. We further adapt VANeRF to the one-shot animatable interacting hands reconstruction. Moreover, We include SMPLpix [8] as an image-space baseline. Besides, although OHTA [18] is not designed for interacting hands reconstruction, we still implement its one-shot strategy with our pre-trained model (denoted as OHTA*) for one-shot performance comparison. Following previous works [13, 8, 15, 18] , we report LPIPS [45] , PSNR [46] , and SSIM [47] as the metrics of rendering quality.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA]Justification: The paper does not include theoretical results or formal proofs.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.• All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
comparison with state-of-the-art methods: 
 Quantitative Comparison. Table 1 reports the quantitative results of our method against the baselines in the one-shot reconstruction scenario, including novel view synthesis and novel pose synthesis. We can see that our method significantly outperforms all methods on all metrics in both tasks. NeRFbased methods, KeypointNeRF and VANeRF fail to have good performance facing large view or pose variations due to the under-utilization of the hand priors. SMPLpix as an image-space method lacks generalization and 3D understanding when coping with single-view reconstruction. Compared with OHTA, our method captures more accurate characteristics of the target identity using the identity map and neural map bias.Qualitative Comparison. Figure 4 demonstrates the visual comparison between our approach and the baselines. SMPLpix fails to produce a reasonable hand appearance with the limited information from a single image. VANeRF predicts the basic hand geometry while leaving high-frequency details like wrinkles and veins. OHTA recovers a reasonable hand geometry with most of the appearance close to the target identity while failing to capture fine-grained identity features. Compared with baselines, our method successfully recovers hand details (e.g. nails, wrinkles, and veins) of the We conduct ablations studies in each of the two stages of our framework. Particularly, we focus on the components of IGSN in the first stage while the loss terms for one-shot fitting in the second stage.The evaluation is conducted in the "train/capture0" subset with 23 pose sequences for training and 1 for testing. The quantitative results are reported in Table 2 Stage-One. The results reflect that each design does bring performance gains and the best performance is obtained by the full model.To validate the effectiveness of our Gaussian points refinement module, we implement a variant by substituting refined Gaussian points with upsampled hand mesh points (denoted as w/o GRM. in Table 2 ). To further verify the effectiveness of our points modification based on the points validation prediction, we implement a variant by replacing the Gaussian points refinement module with simply predicting offset for each point (denoted as w/o GPM. in Figure 5 ).The results indicate that the points modification eliminates the redundant points and densifies the detailed area based on point features, producing better rendering results. Effectiveness of IAttn. Figure 6 demonstrates the predictions of the proposed interaction detection in IAttn. We can observe that our designed interaction detection effectively recognizes the interaction areas, including one-hand self-interaction and the interaction between two hands.To validate the effectiveness of the interaction-aware attention module, we remove it from our full model (denoted as w/o IAttn. in Table 2 and Figure 5 ). The results show that the proposed module improves the reconstruction quality of geometry deformation and texture details caused by interaction based on effective interaction detection.Effectiveness of Identity Map. We also evaluate the effectiveness of the identity map by substituting it with the identity code (vector) as in [18] (denoted as w/o IMap. in Table 2 and Figure 5 ).The results demonstrate that the Identity Map contributes to high-fidelity hand reconstruction by capturing the fine-grained texture of the target identity.We show the effectiveness of our strategies for one-shot reconstruction in Table 2 Stage-Two and Figure 5 . When the identity map is replaced with identity code (IV), the performance drops indicating that our identity map can capture more accurate and fine-grained characteristics of the target hand. When there is no neural texture map bias (III), the details of the target hands are missing. Without color calibration (II), the results become worse due to the color bias. When omitting the identity inversion (I), a significant drop in LPIPS is observed as the target identity information is not captured. Overall, we validate the effectiveness of our design for one-shot reconstruction.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: The setup of experiments is provided in Setup (Sec. 4.1) for reproduction.Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
conclusion: 
 We tackle the challenging single-image interacting hand avatar reconstruction task via an interactionaware Gaussian splatting framework in this paper. Our framework disentangles 3D hand representations into learning-based features that can be extracted by the trained network and identity maps that are one-shot optimized on the hands of new subjects. Additionally, our framework employs an interaction-aware attention module and a self-adaptive refinement module to detect and handle regions with intra-and inter-hand interactions. The proposed method outperforms cutting-edge methods on the Interhand2.6M dataset and creates high-quality avatars for various tasks successfully.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: Code and models will be released upon acceptance.Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/public/ guides/CodeSubmissionPolicy) for more details. • While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark). • The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://nips.cc/public/ guides/CodeSubmissionPolicy) for more details.• The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc. • The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why. • At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable). • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: The training and test details are provided in Setup (Sec. 4.1).Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: The error bars are reported in the appendix (Sec. 6). 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: The information on computer resources is provided in Setup (Sec. 4.1).Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.• The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes]Justification: The research conducted in every respect, with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: The paper discuss both potential positive societal impacts and negative societal impacts of the work performed in the appendix (Sec. 6).Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.• We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The creators or original owners of assets used in the paper are properly credited and the license and terms of use are explicitly mentioned in the Setup (Sec. 4.1).Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA]Justification: The paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained? 
 
 Appendix: A.1 Application.As shown in Figure 7 , We demonstrate more visual results of various applications, including in-the-wild results from real-captured images, text-to-avatar, and texture editing. For text-to-avatar, we utilize ControlNet [48] with depth maps as condition information for hand image generation. For in-the-wild results, we use ACR [43] to estimate hand pose and camera parameters from real-captured images. These results clearly show that our method can be applied to in-the-wild images and obtain considerable results. To evaluate the statistical significance, we run the proposed method five times with different random initiation of identity map in the second stage. Table 3 presents the mean and standard deviation of the results. We use the same captures for training with OHTA [18] in stage one and save the pose sequence '0275_leftbabybird' for testing. For the evaluation of stage two, we conduct experiments on 'test/Capture0' including four pose sequences 'ROM01_No_Interaction_2_Hand', 'ROM01_No_Interaction_2_Hand', 'ROM09_Interaction_Fingers_Touching', and 'ROM09_Interaction_Fingers_Touching_2'.For image preprocessing, we crop out the hand region with the bounding boxes and resize the cropped area to 256 × 256 consistent with previous methods [13, 18] . We adopt SAM [49] to produce hand masks for better segmentation compared to MANO-rendered masks.A.5 More Ablations Ablation study on single-hand images (denoted as Ablation Hand Num.). The advantage of using images of both hands is that two-hand images can provide more complementary information compared with single-hand images, which helps improve the model's performance. We experiment to compare the performance between single-hand images and two-hand images by masking one of the hands from the reference image. The results show that our method achieves better performance compared with the single-hand baseline, which is reasonable as two-hand images contain more complementary information.Ablation study on segmentation method (denoted as Ablation Mask). To validate the impact of segmentation masks, we consider masks predicted by SAM and the ground-truth meshes. We observe that SAM masks enhance the performance of our model, since they are better aligned with hands and prevent background clutters.Ablation S1. Ablation S1 shows the visual results of three different ablation settings: including (i) the use of camera parameters, which is a variant of our method without the camera parameters (w/o Cam.) and suffers from obvious performance drop; (ii) adding shadow coefficient, which incorporates shadow coefficient prediction into the proposed method (w/ Shadow); (iii) Low Gaussian Points, which reduces the number of Gaussian points to 24k to show that coarsen hand boundaries are caused by fewer numbers of Gaussian points.Ablation study on mesh estimation method (denoted as Ablation Noise). We use hand parameters estimated by ACR [43] to analyze how the accuracy of mesh reconstruction impacts the final results. The performance of our method with the predicted parameters is still satisfying, which suggests our method is robust to noise hand meshes to a certain extent. Since OHTA [18] as a single-hand reconstruction method cannot be directly applied to our scenario, we instead use our pre-trained model and only compare the design of the fine-tuning stage. We implement OHTA* with the texture inversion stage and texture fitting stage as described in its paper. In the texture inversion, we keep the network weights frozen and optimize the identity code along with per-channel color calibration coefficients to produce a similar appearance to the target identity of the input image. In the texture fitting, we fine-tune the texture feature MLP for texture feature extraction and constrain the texture-fitting results of some reference views to be close to the rendering results before texture-fitting.Although we have greatly shortened the fine-tuning process, the per-identity training still limits the application compared to single forward inference. Moreover, separate optimization for each identity hinders the model from integrating similar identity information optimized in the fine-tuning stage.When facing serious errors in pose estimation, our method may fail to model the proper appearance through the wrong alignment caused by misleading hand parameters as shown in Figure 10 . Severe estimation errors inevitably cause degraded performance, which is also mentioned in the paper of OHTA [18] .Our method has a positive impact on society as it can facilitate sign language production by creating high-fidelity, animatable hand avatars with interaction. There are no negative societal impacts concerning our work. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.• The answer NA means that the paper does not include experiments. • The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.),(5) where the regularization weight λ reg is user-defined. In our experiments, we find that adding ∆t accelerates the fitting process and helps to prevent undue changes. 


Decision: This is a borderline paper. Based on the initial ratings, and especially after the rebuttal, all reviewers are satisfied with the methodology, evaluation, and the responses provided by the authors.

Reviews: review: paper_summary: paper_summary: - The author extends the concept of one-shot hand avatar creation from the single hand in the previous OHTA paper to two hands.
- The author proposes a novel two-stageGS framework for the reconstruction and rendering of avatars. This framework utilizes 2D identity maps to represent identity information and assist in the learning of neural textures. Additionally, an attention mechanism and a Gaussian refinement module are designed to handle the interaction information.
- The author demonstrates the superior performance of their method compared to the baseline on the Interhand2.6M dataset.
strength_weakness: strength_weakness: - The article represents an incremental work.

- Initially, it is not convincing why one would construct a hand avatar using images of both hands with interaction instead of building them separately. What are the advantages?

- Secondly, the advantage of the one-shot method lies in addressing in-the-wild scenarios. The pioneering work in the single hand direction, OHTA, demonstrated a multitude of in-the-wild results, but this paper does not. This raises doubts about the effectiveness of this method in in-the-wild scenarios.

- The method's modeling of the hand is not sufficiently detailed. Previous methods such as Handavatar, OHTA, etc., have considered shadows caused by occlusions, but it is not clear how this article's method learns such priors.
questions: questions: - The OHTA* method compared in the article seems more like an ablation of the full model. The performance of the OHTA algorithm reproduced by the authors is worse than in the original paper. Can an explanation be provided on how this baseline was implemented, and can some reasons for the poorer performance be analyzed?

- The paper incorporates camera intrinsic and extrinsic parameters as part of the Geometric Encoding. What are the advantages of introducing this information? Is there any related ablation analysis?

- Is the initial mesh of the bimanual results shown by the author estimated or taken from the dataset's ground truth? If it is estimated, what estimation method was used? How is the error in pose estimation addressed?
limitations: limitations: The author has already addressed the limitations of the method.
paper_summary: paper_summary: This paper proposes a novel two-stage interaction-aware GS framework to create animatable avatars for interacting hands from single-image inputs. The proposed method disentangle the 3D presentation of hands into optimization-based identity maps and learning-based latent geometric features and neural texture maps, and exploits cross-subject hand priors and refines 3D Gaussians in interacting areas. Experiments are conducted on the Interhand2.6M dataset with state-of-the-art method comparisons.
strength_weakness: strength_weakness: 1) The data flow in Figure 3 is somewhat unclear. For example, what does the dotted line from the Interaction Points to the Interaction-Aware Attention present? In addition, using different colored sub boxes for geometric flow and texture flow can help readers more intuitively understand the disentanglement of hand priors.
2) In line 119 of page 4, there is a typo error. 'Sec.' is missing.
3) The generalization of in-the-wild images has not been demonstrated.
questions: questions: 1) In lines 85-87 of page 3, why can disentangling hand prior learning reduce the cost of one-shot fitting？Is this related to introducing texture map bias to accelerate one shot fitting
2) Why is the interacting label $d(q)$ binarized, rather than applied by soft activation?
3) An important baseline, OHTA, was re-implemented and compared in the experiment. Can you provide more details?
limitations: limitations: Yes
paper_summary: paper_summary: The paper proposes an approach to achieve one-shot interacting hand avatar reconstruction via 3D Gaussian Splatting. The authors design a two-stage framework: the first stage learns learning-based features and optimization-based identity maps, and the second stage performs one-shot reconstruction by optimizing only the identity map. To better capture the relationship between the two hands, they devise an interaction-aware attention module. Moreover, they design a self-adaptive GS density control approach for this task. The experiments demonstrate the effectiveness of their designs and show that the approach achieves state-of-the-art performance.
strength_weakness: strength_weakness: 1. For interacting hands, significant shadows occur between the two hands. The proposed method does not account for shadow modeling, which overlooks the dynamic nature of the hands and makes the approach less practical for real-world scenarios to some extent.
2. The experimental results are not comprehensive.
- It would be beneficial to present some in-the-wild testing results to demonstrate the method's plausibility for real applications. This would also justify the design for addressing out-of-distribution (OOD) data.
- Including some failure cases would help to illustrate the boundaries of the work.
questions: questions: 1. Why should we use interacting hands for avatar creation? What are the benefits compared to single-hand modeling?
2. What is the detailed implementation of OHTA? The results appear to be worse than the reported results.
3. How does the accuracy of mesh reconstruction impact the final results?
limitations: limitations: The authors present the "Limitations and Society Impact" section.
paper_summary: paper_summary: This paper proposes a method to create a 3D Gaussian Splatting (GS)-based avatar for interacting hands from single-image inputs. Its main contribution is to propose a two-stage GS framework (1) to leverage cross-identity priors via learning-based features and also to (2) well preserve per-identity information via optimized identity maps. Additionally, it proposes an interaction-aware attention module and a Gaussian refinement module to further exploit the hand interaction context for more realistic rendering. In the experiments, the proposed methods outperforms the existing methods based on generalizable and one-shot NeRFs.
strength_weakness: strength_weakness: **(1) Effects of off-the-shelf MANO regressor and mask detector.** The proposed avatar reconstruction method (Sec. 3.3) uses the outputs of off-the-shelf MANO regressor and segmentation model. It would be informative to include the discussions on how the quality of these outputs might affect the final rendering quality.

**(2) Non-smooth hand boundaries in rendering results.** In Fig. 4, I’ve noticed that the boundaries of hands rendered using the proposed method are less smooth compared to the existing methods. Does it indicate that the proposed GS-based method (implicitly) learns less smooth hand geometries than the other NeRF or mesh-based methods? More discussions regarding this would be informative.
questions: questions: (1) In lines 94-95, the paper claims that the rendering quality of the existing MANO-based methods “is constrained by the coarse geometry and sparsity of the MANO mesh”. However, to my knowledge, HARP [3] and Handy [5] use higher resolution-version of MANO meshes to already address this issue. What would be the advantage of using GS compared to using such higher-resolution hand meshes?

(2) In lines 118-119, the paper states that the proposed method "reduce[s] the time consumption of one-shot avatar reconstruction". Have you tried comparing the reconstruction time of the proposed method against OHTA [18]?
limitations: limitations: The authors discussed the limitations in Sec. A.4 in Appendix.

Comments: title: Our Responses to Questions
comment: Dear Reviewer mjNr,

Thank you for your appreciation. Your suggestions and feedback truly contribute to the improvement of our paper.  We will ensure that all revisions are included in the final version.
I appreciate the authors' efforts to address my concerns. Most of my questions have been resolved, so I raised my score.
Dear Reviewer gWjw,

Thank you very much for your time and appreciation. All the revisions (in-the-wild qualitative results and detailed comparisons) will be included in our revised paper.
Dear Reviewer eWJx,

Thank you very much for your appreciation. Your suggestions and comments indeed help to improve our paper greatly. We will include more visual results (especially for the in-the-wild images) and all other revisions in our paper.
Dear Reviewer NUjU,

Thank you for increasing the score of our paper.  All the revisions will be incorporated into our paper. The link to the code and models will be provided in our revised paper to facilitate the following research.
I appreciate the authors' efforts in responding to the feedback. While the response addresses some of my concerns, I continue to suggest that the authors present more visually appealing results in the revised version to better demonstrate their method's effectiveness. Furthermore, I strongly recommend that the authors incorporate the suggestions from all reviewers to enhance the overall completeness of the paper. Based on the current revision, I am inclined to rate this paper as Borderline Accept.
I appreciate the authors' proactive response. The supplementary ablation experiments on camera parameters and the additional experiments using predicted meshes instead of GT meshes have addressed my concerns.

Regarding OHTA and the authors' reproduced versions OHTA* or OHTA**, I suggest that the authors clarify their descriptions in the revision. Based on my understanding, the actual OHTA is not equivalent to OHTA* (Inv+Calib), as there are other details such as the use of reference views for assistance during finetune.

I'd also like to gently remind the authors about their commitment in the "Open access to data and code" section, where they mentioned that "Code and models will be released upon acceptance." Following through on this would be greatly appreciated and could help address any lingering concerns about reproducibility that reviewers might have.

At this point, I am inclined to give this paper a score of Borderline Accept.
Dear Reviewer eWJx,

Thanks for suggesting improving the representation of shadow modeling for in-the-wild images. Although shadow modeling is beyond the scope of this paper (as we don't have ground-truth albedo images for explicit supervision), we provide additional examples of this in the following anonymous link: 

https://anonymous.4open.science/r/in_the_wild_shadow-35EB/README.md.

Please have a look at these examples, where we show in-the-wild inputs (left) and the synthesized images (right-top), albedo images (right-middle), and shadow images (right-bottom) under different poses. We also label shadow areas caused by occlusions in red rectangles, which clearly show that our method generates appealing interacting shadows of interacting hands.  

We sincerely hope this can address your concern and help to improve your rating of our paper.
The author provided the in-the-wild qualitative results, and more detailed comparisons of the baseline method OHTA, which to some extent addressed the previous concerns. Therefore, I decide to maintain the initial rating, Weak Accept.
Thank you for your response. I appreciate the authors' efforts in addressing my concerns, but they remain unresolved.
Regarding shadow modeling, while the authors demonstrate the proposed method's ability to model shadows in the InterHand2.6M dataset, they have not shown its effectiveness in capturing interacting shadows in in-the-wild testing data. I recommend that the authors present visually appealing examples of interacting shadows in these in-the-wild results.
Additionally, the in-the-wild visuals are not convincing. I suggest using higher-quality images and improved presentation methods to better showcase the results.
Dear Reviewer mjNr,

We have tried to address your concerns in our earlier responses. We are looking forward to your suggestions. May we know if our rebuttals answer all your questions? We truly appreciate it.
Dear Reviewer eWJx,

Thank you for taking the time to provide valuable suggestions. We have made our best efforts to address your concerns. If you have any other questions or suggestions, we are very grateful to discuss them with you.
Dear Reviewer gWjw,

We are grateful for your previous comments and suggestions. We have responded to your comments in earlier days. If you have any other questions or comments, we will be very happy to discuss them with you.
Dear Reviewer NUjU,

We have tried our best to address all the concerns in our earlier responses. We are very happy to discuss any additional questions or suggestions you have.

Thank you for your time and consideration.
**Q3 The impact of the accuracy of mesh reconstruction**: We conduct an experiment to estimate the effects of different meshes. In this experiment, we compare the performance of our method with the ground-truth meshes provided by the Interhand2.6M dataset, and that with meshes predicted by ACR. The qualitative results are shown in RF-Fig. 2 Ablaiton Mesh and the quantitative results are listed as follows:

| Method | Mesh |   PSNR    |   LPIPS   |   SSIM    |
| :----: | :--: | :-------: | :-------: | :-------: |
|  Ours  |  GT  | **26.14** | **0.869** | **0.161** |
|  Ours  | ACR  |   25.83   |   0.864   |   0.167   |

The performance of our methods with parameters estimated by ACR is still satisfactory. This is because we have added regulation to texture map bias to prevent our model from over-fitting the incorrect alignment between hand appearance and geometry, which addresses minor estimation errors effectively.
**Q1 Implementation of OHTA***: Please note that OHTA is a single-hand reconstruction approach, and **its setup and dataset completely differ from ours**. Because its pre-trained model cannot be used in our scenario, we utilize our own pre-trained model and only compare the fine-tuning stage design. The code of OTHA is not published before the deadline for paper submission, hence we implement OHTA* with the texture inversion stage and texture fitting stage following its paper. In the texture-inversion stage, we keep the weights of the pre-trained network frozen and optimize the identity code along with per-channel color calibration coefficients to fit the input image. In the texture-fitting stage, we fine-tune the MLP for texture feature extraction and impose the constraint that the texture-fitting results of reference views are close to the rendering results before texture-fitting.

Since our fine-tuning process only contains one stage while OHTA contains two fine-tuning stages, we use twice the training steps for OHTA* (one for each stage). Concerning the inferior performance of OHTA*, we consider that the previous number of training steps might be insufficient for OHTA* to fully fit the reference image. To investigate this, we further prolong the training steps (denoted as OHTA**) and obtain better performance as follows:

| Method |   PSNR    |   SSIM    |   LPIPS   | Fine Tuning Time |
| :----: | :-------: | :-------: | :-------: | :--------------: |
|  Ours  | **26.14** | **0.869** |   0.161   |   2.5 minutes    |
| OHTA*  |   25.31   |   0.851   |   0.184   |    5 minutes     |
| OHTA** |   25.96   |   0.864   | **0.160** |    25 minutes    |

Yet, our method still achieves a better balance between rendering quality and fine-tuning time. 

**Q2 The advantages of introducing camera parameters**: Thank you for your advice. We conduct the related ablation experiments to analyze the impact of the camera parameters. The qualitative results are shown in RF-Fig. 2 (denoted as w/o Cam. in Ablaiton S1) and the quantitative results (denoted as w/o Cam.) are listed as follows:

|  Method  |   PSNR    |   SSIM    |   LPIPS   |
| :------: | :-------: | :-------: | :-------: |
|   Ours   | **28.11** | **0.902** | **0.130** |
| w/o Cam. |   25.91   |   0.862   |   0.198   |

The performance of the baseline without camera parameters drops obviously. Considering the anisotropy properties of Gaussian points (e.g. scale and opacity), it is necessary to utilize camera parameters to encode the view direction information for rendering.

**Q3 Mesh estimation method**: We use the ground-truth hand parameters provided by the dataset. We conduct the experiment using hand pose parameters estimated by ACR. The qualitative results are shown in RF-Fig. 2 Ablation Mesh and the quantitative results are listed as follows:

| Method | Mesh |   PSNR    |   LPIPS   |   SSIM    |
| :----: | :--: | :-------: | :-------: | :-------: |
|  Ours  |  GT  | **26.14** | **0.869** | **0.161** |
|  Ours  | ACR  |   25.83   |   0.864   |   0.167   |

The performance of our methods with parameters estimated by ACR is still satisfactory. This is because we have added regulation to the texture map bias in order to prevent the model from over-fitting to the incorrect alignment relation between hand appearance and geometry, which addresses minor estimation errors effectively. However, as demonstrated in the failure cases in RF-Fig. 3, serious estimation errors make it difficult for our model to learn the correct appearance. Previous works like OHTA also mention this issue, which might be alleviated by more reliable hand regressors.