Learning Interaction-aware 3D Gaussian Splatting for One-shot Hand Avatars
Authors: Xuan Huang
Hanhui Li
Wanquan Liu
Xiaodan Liang
Yiqiang Yan
Yuhao Cheng
CHENQIANG GAO
Venue: NeurIPS 2024 poster (2024)
Keywords: Gaussian splatting
interacting-hand avatar
one-shot learning
Primary area: machine_vision

Abstract: In this paper, we propose to create animatable avatars for interacting hands with 3D Gaussian Splatting (GS) and single-image inputs. Existing GS-based methods designed for single subjects often yield unsatisfactory results due to limited input views, various hand poses, and occlusions. To address these challenges, we introduce a novel two-stage interaction-aware GS framework that exploits cross-subject hand priors and refines 3D Gaussians in interacting areas. Particularly, to handle hand variations, we disentangle the 3D presentation of hands into optimization-based identity maps and learning-based latent geometric features and neural texture maps. Learning-based features are captured by trained networks to provide reliable priors for poses, shapes, and textures, while optimization-based identity maps enable efficient one-shot fitting of out-of-distribution hands. Furthermore, we devise an interaction-aware attention module and a self-adaptive Gaussian refinement module. These modules enhance image rendering quality in areas with intra- and inter-hand interactions, overcoming the limitations of existing GS-based methods. Our proposed method is validated via extensive experiments on the large-scale InterHand2.6M dataset, and it significantly improves the state-of-the-art performance in image quality. Code and models will be released upon acceptance.

Learning Interaction-aware 3D Gaussian Splatting for One-shot Hand Avatars
Xuan Huang' Hanhui Li";
Wanquan Liu', Xiaodan Liang',
Yiqiang Yan?, Yuhao Cheng", Chenggiang Gao Shenzhen Campus of Sun Yat-Sen University
Lenovo Research
huangx3556mail2. wys.........u.c..... edu. cn 1ihh77@mail
1iuaq630mail 1. sysu. wttp:300mmm......dduuuuu..o en xcliaang328gmmil.comm
vanyq@lenowe com chengyh5@lenovo.. com
gaochg66mail edu.
Abstract
In this paper, we propose to create animatable avatars for interacting hands with 3D Gaussian Splatting (GS) and single-image inputs. Existing GS-based meth- ods designed for single subjects often yield unsatisfactory results due to lim- ited input views, various hand poses, and occlusions. To address these chal- lenges, we introduce a novel two-stage interaction-aware GS framework that exploits cross-subject hand priors and refines 3D Gaussians in interacting eas. Particularly, to handle hand variations, we disentangle the 3D presentation of hands into optimization-bast metric features and neural texture maps. Learning bearning-based features are captured by trained networks to provide reliable priors for poses, shapes, and textures, while optimization-basee based identity maps enable efficient one-shot fitting of out-of of- distribution hands. Furthermore, we devise interaction-awe attention module and self-adaptive Gaussian refinement module. These modules enhance image rendering quality in areas with intra- and inter-hand interactions, overcoming the limitations of existing GS-based methods. Our proposed method is validated via extensive experiments on the large-scale InterHand2.61. dataset, and it signifi- cantly improves the state-of-thee the-art performance in image quality. Project Page: https: //github//tt com Xwwnnuaang// Sutilaaannngg/assaiiiaaaa anHand
identity maps and learning-base based latent geo-
Introduction
Recent advancements in 3D reconstruction and differential rendering techniques have significantly improved hand avatar creation and related applications. However, creating avatars for interacting hands from a single image remains challenging. The limited input view does not provide sufficient geometry and texture information for accurate reconstruction. Moreover, intra- and inter-hand interactions exacerbate information loss and introduce complex geometric deformations.
Extensive efforts have been made to tackle these issues, shown in Figure 2 (a) Early approaches depend on explicit parametric meshes (e.g. MANO (T] for geometry modeling, and utilize UV map 2 3 4 5 vertex color [6 7. image space rendering [8] for appearance. Despite the
efficiency in rendering, these methods fail to achieve realistic rendering results with the coarse
mesh resolution and the simple combination of hand appearance and geometry. (b) More recently,
"Both authors contributed equally. "Correpoonding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Animatable Avatar
Single Image
Rendering with Novel View
light
Color Interpolation
"kand,
Editing
Hand Combination
Editing
Figure 1: We present novel interaction-awarr Gaussian splatting framework that creates animatable interacting hand avatars from a single image. These high-fidelity avatars support various applications, such editing, animation, combination, duplication, re-scaling, and text-to-avatar conversion.
with the significant success of neural radiance fields (NeRF), extensive studies 9 10
12
usually require per-seene optimization for each new identity using densely calibrated images, which results in expensive training costs. Generalizable NeRFs [14] [5] [3 [6] get rid of per-scene training by leveraging imagc-aligned features to enable reconstruction from a few or even a single view. Yet their dependence on image-aligned features also limits their performance under large view pose variations. Besides, (c) One-shot NeRF- methods 17 18 propose to exploit data-driven priors with condition optimizations [17 and inversions [8]. Nevertheless, these methods
have employed NeRF-based models for implicit modeling. These methods [00
suitable for our task, as they do not include any module to detect and handle interactions. Moreover, the inversion of identity vectors used in [8 omit the spatial image structure, which not only hinders its performance but also introduces extra time consumption for fine-tuning networks.
To tackle the above issues in existing methods, aim to create animatable avatars for interacting hands with 3D Gaussian Splatting (GS) and single-image inputs. To this end, we introduce a novel two-stage interaction-awart GS framework shown in Figure (d). We disentangle the 3D presentation of hands into (i) features that can effectively captured by training networks in the first stage of our framework (e.g., geometric features and latent neural texture maps), and (ii) identity maps that can be optimized efficiently in the per-subject one-shot fitting stage. In this way, our method not only enables leveraging cross-subject priors with learning- peaniig-bbseed based features, but also well preserves per-subject characteristics via optimizing identity maps.
Conditional
Generator d
(d)
Discntangled Hand
Stage-I:
Interatioss-oout Splating Neework
Cadiiies
Sepervine
(b)
Differcntabbee Renderer
Enceder
Tronee Credklons Neara Tinbas Mago
Zped Image
Resterad truge
Traazs laagee
Coa-
Irage
Ganeilan Splatting
Superviee
Sapervioe
Refreenco
hand
taversion
Mudula
Diffcreatisble Randeer
Partial
Modula $
Partul
Supmrii::
IGSN
Manty
Iruge
Rativmnc
Ilawil Mar
Target
Slap Bisa
Baaddred
Figure 2: Paradigm comparison between existing one-shot hand avatar methods (a-c) and the proposed method (d). By decoupling the learning and fitting stages, our method leverages the advantages of learning-basse methods (a, b) in modeling cross-subject hand priors, and the advantages of inversion- based methods (c) in one-shot fitting without the extra cost network fine-tuning.
Additionally, to achieve robust reconstruction and enhance rendering quality, we devise an interaction- aware attention module and a self-adaptive Gaussian refinement module. The former module identifies Gaussian points with potential intra- and inter-hand interactions to enhance their features

---

with attention mechanisms. This enhancement allows our GS network to better model geometric deformations and fine- auee-rained textures caused by interactions (e.g. wrinkles). The latter module is introduced to address the limitations of the coarse geometry of parametric hand meshes, which is achieved by learning to eliminate redundant Gaussians and assigning extra Gaussians in regions with complex textures and deformations. Consequently, our method can reconstruct realistic inter-hand avatars with great flexibility for animation and editing, as shown in Figure]
Overall, our contributions can be summarized as follows:
We propose a novel two-stage interaction-aware GS framework create animatable avatars for interacting hands from single-image inputs. Our method generates high-fidelity rendering results and supports various applications. Experimental results on the large-scale Interhand2.60 6M dataset [19] validate the superior performance of our method compared to previous methods.
We disentangle the 3D presentation of hands into learning- basedd features that can be generalized well to different subjects and identity maps that are individually optimized for each subject. This disentangiement provides us with flexible and reliable priors for poses, shapes, and textures.
We introduce an interaction-aware attention module, which identifies intra- and inter-hand interac- tions and further exploits interaction context to improve rendering quality.
We devise Gaussian refinement module that adaptively adjusts the number and positions of 3D Gaussian, which results in rendered images of higher quality under various hand poses and shapes.
2 Related Work
One-shot Human Reconstruction. One-shot reconstruction of 3D humans is challenging and long- standing problem due to the limited information from a single input image. To alleviate this limitation, previous works leveraged parametric models [20 [ as coarse geometry prior. Traditional methods
25) utilized UV maps for human appearance representation. To complete the unseen texture from the single image, some approaches [25] 24 inpainted missing textures via pre-trained diffusion models. Neural Radiance Fields (NeRF) (26) have also been explored for reconstruction
21 22 23
from sparse views (27 15 [14 28 or single view (29 16 13 18). KeypointNeRF [15] encoded spatial information using 3D skeleton keypoints. SHERF [6 created 3D human avatars from a single image with hierarchical features for informative encoding. VANeRF [13 leveraged visibility in both feature fusion and adversarial learning for single-view interacting
handd hand image synthesis. These methods greatly enhance the NeRF one-shot reconstruction performance. Nonetheless, generalizable [4| [5 fails to achieve satisfactory results when the input image information is not sufficient for novel view prediction due to the under-utilization of hand priors. To overcome this issue, the pioneering research in [18] enabled one-shot single-hand avatar creation by learning data-driven hand priors which are further utilized with inversion and fitting. Compared with previous methods, our approach further disentangles the presentation of hand priors into latent geometric features, neural texture maps, and optimizable identity maps to enhance rendering quality and reduce the cost of one-shot fitting. Moreover, we introduce the interaction-awe aware module and self-adaptive refinmeent module, which helps to significantly improve the visual quality of synthesized images.
NeRF (27 30 31
Animatable Hand Avatar. Conventional methods created hand avatars by incorporating UV textures with explicit parametric hand models. HTML [2] is the first parametric texture model of human hands which models hand appearance with several dimensions of variability. HARP 3 further introduced albedo and normal information into UV maps to represent hand appearance without any neural components. Handy [5] realistically captured high-freuennc detailed texture using a GAN- based texture model. However, the rendering quality of these methods constrained by the coarse geometry and sparsity of the MANO mesh. The recent ad advancmennss in the neural radiance field have resulted in the development of approaches that utilize implicit representation for hand reconstruction. LISA [9] is the first method that employs NeRF to learn the implicit shape and appearance of hands. HandAvatar [I] developed a high-resolution variant of MANO to fit personalized hand shapes and further disentangled the implicit representations for hands into geometry, albedo, and illumination. HF (10) designed a pose-driven deformation field with pose-disentangled
NeRF to reconstruct single or interacting hands from multi-view images. LiveHand [12] proposed a low-resolution rendering of NeRF together with a super-resolution module to achieve real-time performance.
3

---

Ideasit Eneader
Feature Seattering
Testure Beeeder
Coas-ddec lerrnty Nips
ldratty Trinddiigs
Tannee
Nearal Tequre Maps
Mesh Recontraction
Pose Eaceder
Frarore Sampling
Gaussian Refiemenn Module
Ganssiaa Decoder & Splatting
Uibbedding
Interaction Detection
Goometry Decoder
Interaction-Aww Atteation
Techur Phon
Iskancingg
Locreable
Pacitional
Rankred Imaage
Fnesder
Poctioal Rriweklegs
Interaction-Awaal Gaussian Splatting Network
Figure 3: The architecture of the proposed interaction-aware Gaussian splatting network, of which the core components are the disentangled hand representation, the interaction detection module, the interaction-aware attention module, and the Gaussian refinement module labeled in green.
Point-based 3D Representation and Rendering. There has been a growing interest in point-based neural rendering due recent advancements in 3D Gaussian splatting (3DGS) [32] which has led to high-quality and real-time rendering speed. Since 3DGS is primarily designed for static scenes, dedicated expanding its applicability free pose animation and rendering. 3D-PSHR 34 achieved real-time time and photo-realistic hand reconstruction from large-scale multi-view videos based on 3D points splatting. Our method differs from 3D-PSHR as it allows instant single-view one-shot hand avatar reconstruction and saves the expensive computational cost of per-scene optimization.
many efforts 33
34
35
36
Methodology
In this section, we introduce the details of the proposed two-stage framework for creating interacting hand avatars from a single image. The key ideas of our framework contain three aspects: (i) We address the lack of information caused by limited inputs by learning disentangled priors for hand poses, shapes, and textures (Sec. 3.T). (ii) We construct an interaction- Gaussian splatting network handle both intra- and inter-hand interactions (Sec. (.2)). (iii) Leveraging invertible identity and neural texture maps, we reduce the time consumption of one-shot avatar reconstruction while simultaneously improving the quality of synthesized images (Sec. 3.3)
3.1 Disentangled 3D Hand Representation
To demonstrate the motivations of the proposed disentangled 3D hand representation, we first provide the formulation our task.
Task formulation. Given a reference image of interacting hands 4,,, our task is to reconstruct an animatable two-hand avatar that can generate images of the hands with novel poses and from novel views. To achieve this, propose to construct differentiable renderer R x| R #xxWWx x where 0 and denote the hand pose parameters and camera parameters, respectively. H and W denote the height and width of rendered three-channel RGB images.
More specifically, we propose to implement R via Gaussian splatting (GS) 32 for its advantages in explicit modeling and computational efficiency. Essentially, GS is point-based rasterization technique that leverages set 3D points (Gaussians) with attributes like colors, opacity, and spherical harmonic coefficients represent reference images. Let P R. denote the set of N Gaussians and a E R x denote their D-dimensional attributes. To preserve the characteristic of the hands in I,,, need to optimize R via minimizing the following 4 loss,
arg min R((,c (P.a) 1-ll,). P.a,0.c
(1)
Due to the high-dimensional attribute space and inter-attribute interference, direct optimization for Eq. I is intractable. Although learning-based methods can alleviate this issue to certain extent, their

---

results may fall short if I, lies outside of the distribution (OOD) of their training data. Therefore, a disentangled representation that integrates optimization- based and learning-basee Gaussian attributes is essential for addressing our task effectively. Furthermore, a reliable representation should capture both the geometric and textural properties of hands. Given the availability of extensive hand mesh reconstruction methods for geometric information, we propose leveraging learning- sarning-based geometric properties alongside optimizable textures in our disentangled representation. This approach allows us to handle the diversity and potential OOD issues of hand textures.
To this end, we devise the disentangled representation shown in Figure 3 to combine explicit geometric embeddings from hand meshes with neural texture maps encoding implicit latent fields. The encoders for this representation are learned on training dataset consisting images of S subjects. Specifically, we first construct the optimizable cross-subject identity maps m E R 88x20'x x where C denotes the number of feature channels of one hand. For each training image I, we reconstruct parameterized hand mesh M from it and use the vertices of M to initialize P. Let E [1.... S] denote the subject ID of I,, we retrieve corresponding identity map m, and combine it with the pose embedding from MM to infer its neural texture map t, E =22**x Consequently, Vp E P. we can query their feature vectors from tg based on their texture coordinates and predict their Gaussian attributes. In the one-shot fitting stage, to obtain the neural texture map of I,, we only need to optimize a new identity map initialized with zeros. This is applicable, as our identity maps and natural texture maps share important advantage: they both preserve the spatial structure of textures, which overcomes the burdens of previous vector- inversion methods [8]
Below we introduce the core components for implementing our disentangled representation. These components will be integrated into the GS network in the next section for end-to-end learning.
Parameterized Hand Mesh. We employ MANO [T] to reconstruct hand meshes from images for its convenience in animation. MANO is a parametric model that represents hand mesh by pose parameters 0 E R 48 and shape parameters 3 E IR_.. To further improve the mesh quality, we use high-resolution version of MANO I.. We follow [22 to obtain normalized UV coordinates (u, v) E [0. 1] x [0, 1] of mesh vertices and project them onto the neural texture plane.
Geometric Encoding. To exploit explicit geometric features from hand meshes, utilize a pose encoder and a positional encoder. The pose encoder is an MI. taking the pose parameters 0 and the camera parameters e 2225 inputs (which is the flattened concatenation of an extrinsic matrix E 4 and an intrinsic matrix E TR5x3). Our pose embeddings do not involve /3 and hence they are independent of identities. The positional encoder is a shallow PointNet (38] with local pooling (39) We further employ a transformer-based decoder to merge the outputs of the pose encoder and the positional encoder, similar
440)
Texture Encoding. Given the UV coordinates of a vertex, we retrieve its feature vector on the optimizable identity map, along with the Y positional encoding (26) of its coordinates to generate its identity embedding. The identity embeddings of all vertices are concatenated with the pose embedding and projected (scattered) back to the UV plane to form texture condition map. We again adopt a transformer- based decoder to process the condition map and yield the neural texture map.
Finally, we combine the geometric feature vectors of all vertices with their texture feature vectors using element-wise, addition. This results in unified latent representation f E IR I xC, which we use for predicting Gaussian attributes.
3.2 Interaction-Aware Gaussian Splatting Network
To better reconstruct interacting hand avatars with various poses, we propose to enhance the Gaussian features f via an interaction-awan awwrre attention (IAttn) module and a Gaussian point refinement module (GRM). IAttn identifies potential points with intra- or inter-hand interaction. By exploring the context around interaction points, IAttn improves the reconstruction quality of geometric deformations and texture details resulting from interactions (such as shading, wrinkles, and veins). Furthermore, GRM not only eliminates redundant Gaussians but also generates additional Gaussians near regions with complex textures. With these two modules, our network can render high-quality hand images with rich details.
5

---

3.2.1 Interaction-aware Attention
To detect interacting points in P, we propose straightforward yet effective strategy that calculates the difference between the neighboring point sets posed hand meshes and canonical mesh. This strategy is practical arbitrary query point E P. if its top-N, nearest neighboring points on the canonical mesh (g) significantly overlapped with its top-Np nearest neighbors the posed mesh (denoted as S2p), the chance that is an interacting point is low. This strategy can be formulated as follows,
define interaction-free mesh the canonical one. For
1, if (2(g) U&p(g) Pe(g) n (2(()) T, 0, otherwise,
d( =
where T is a user-defined threshold. Additionally, we append the interacting label d(q) to the pose embedding introduced the last section. Note that our proposed strategy can detect both self- interacting and cross-interacting points. To maintain the efficiency of our method, we only conduct self-attention on detected interacting points.
3.2.2 Self-adaptive Refinement for 3D Gaussians
Gaussians initialized from MANO can only provide coarse hand geometry with restricted defor- mations. To better model the geometry of hands with various poses and shapes, we devise the self-adaptive GRM to control the density of Gaussians and refine their locations. Given a Gaussian point p and its corresponding feature vector fp. we utilize an MLP o(f,) with the sigmoid activation to predict the validity of i.e., Φ C - [0, I]. We remove from P if ((f,)) (,) is below pre-defined threshold Ta while splitting p ((,,) is larger than another threshold T,- We also exploit f to predict the offsets of Gaussians to adjust their positions.
3.2.3 Network Optimization
With the search space of Gaussian attributes reduced significantly by the proposed disentangled representation, we are now ready to optimize our GS network and learn cross-subject hand priors. We train the GS network along with the optimizable cross-subject identity maps m by minimizing the following loss:
1= R(0, c, m, P(O), Lree = Argb || Tell + AvGGLvGG (I, Tt),
(3)
where are the learnable parameters in our GS network. LVGG denote the perceptual loss 41 Angb and λvGG user-defined weights.
3.3 One-shot Hand Avatar Reconstruction
In the stage of one-shot hand avatar reconstruction, the parameters of IGSN are fixed and we fine-tune the identity map the new subject m E R2C x xxW" This can be formulated as,
arg minCino. Lree (R(Or, m", P.(0), + Amask || M.m
Mr
where the geometric parameters 6,, Cr, and P, can be obtained via off-the-shelf MANO regressors 42 43). M and M, denote the hand mask of I and 4,, respectively. Amask is the empirical weight of the loss term on masks.
18 suggests that optimization tricks like color calibration and view regularization can further improve the synthesized images. Inspired by this, we also introduce a texture map bias At E C Hx to modulate the latent neural feature maps of two hands (t {t,, )) That is,
+ At and t, = f, + At. We assume that At can be shared by t, and t, due to the symmetry of the left and right hands. We include a regularization term At into Eq. (4) to prevent drastic shifts of t as follows:
arg min ( (Cine. + Areg | 1 M.m ,At
(5)
where the regularization weight Areg is user-defined. our experiments, we find that adding At accelerates the fitting process and helps to prevent undue changes.

---

Table 1: One shot synthesis comparison with state-of-the-art methods on Interhand2.6M.
Novel View Synthesis
Novel Pose Synthesis
Method
PSNRT
SSIM
LPIPST
PSNRT
SSIM
LPIPST
KeypointNeRI
23,55
0.804
0.326
SMPLpix
24.50
0.868
0.170
24.26
0.854
0.173
VANeRF
25.38
0.848
0.226
24.42
0.822
0.250
OHTA*
25.31
0.851
0.184
25.93
0.880
0.156
Ours
26.14
0.869
0.161
26.56
0.890
0.133
4 Experiments
4.1
Setup
Learning IGSN. Our experiments are conducted on the publicly available Interhand2.cm 226 dataset (CC-BY-NC 4.0 licensed) that consists of large-scale multi-view sequences of different subjects pose sequences of 21 subjects from InterHand2.61 training set for pre-training. For each subject, unseen pose sequence used for evaluation. Our network is trained on three A6000 GPUs using the Adam optimizer 44] with the learning rates of 10-4 for eight epochs. Loss weights in Eq. (3) set as Argb = 10.0, λyGG = 0.1. For interaction detection, we set N = 100 and T = 90. For self-adaptive GRM, set Ta 0.1 and T, 0.9. We adopt coarse-to-fine mesh refinement strategy during training: For the first 5 epochs, we upsample hand meshes to 12,337 points per hand while for the last three epochs, further upsample hand meshes 49,281 points per hand.
19
performing various hand poses. Following [8], we adopt interacting
One-shot Reconstruction. We conduct one-shot reconstruction evaluations on the testing set of InterHand2. 6M as in [18 [I]. To evaluate the novel pose rendering quality, We evenly sample 349 frames from four pose sequences including four common views in the "test/captureO" subset as the test set. To assess the quality of novel view synthesis, the "test/captureO" subset. The one-shot fitting takes 50 optimization steps with the learning rate of 1 X 10 The whole process takes 2.5 minutes with an A6000 GPU. Loss weights in Eq. 44 are set as Amask = 1.0, Areg = 0.01.
have selected the initial 50 views from
We select four state-of-the-ard methods for comparison. We adopt two generalizable NeRFs including KeypointNeRF [15] designed for human novel view synthesis and VANeRF [13 designed for single-view interacting-han -han image novel view synthesis. We further adapt VANeRF to the one-shot animatable interacting hands reconstruction. Moreover, We include SMPLpix 8 as image-space baseline. Besides, although OHTA [18] is not designed for interacting hands reconstruction, we still implement its one-shot strategy with our pre-trained model (denoted as OHTA*) +) for one-shot performance comparison. Following previous works
Baselines and Metrics.
[13 8
[5
[8]
report
LPIPS 45 PSNR 46). and SSIM[47] as the metrics of rendering quality.
4.2 Comparison with State-of-the-art Methods
Quantitative Comparison. Table/[rppor the quantitative results of our method against the baselines in the one-shot reconstruction scenario, including novel view synthesis and novel pose synthesis. We can see that our method significantly outperforms all methods on metries both tasks. NeRF- based methods, KeypointNeRF and VANeRF fail to have good performance facing large view pose variations due to the mnder-utilization of the hand priors. SMPLpix as an image-space method lacks generalization and 3D understanding when coping with single-view reconstruction. Compared with OHTA, our method captures more accurate characteristics of the target identity using the identity map and neural map bias.
Qualitative Comparison. Figure demonstrates the visual comparison between our approach and the baselines. SMPLpix fails to produce reasonable hand appearance with the limited information from a single image. VANeRF predicts the basic hand geometry while leaving high-frequency details like wrinkles and veins. OHTA recovers reasonable hand geometry with most of the appearance close to the target identity while failing to capture fine-grained identity features. Compared with baselines, method successfully recovers hand details (c.g. nails, wrinkles, and veins) of the

---

GT
SMPLpix
VANeRF
OHTA*
Ours
GT
SMPLpix
VANeRF
OHTA*
Ours
Qualitative comparisons with state-of-the-an methods. The input image is shown in the top-left grid labeled in red. The first row presents results without changing the pose from the input view (left) and alternative view (right), while results in the remaining rows are with novel poses.
Figure 4:
target identity for various views and poses. The qualitative comparisons demonstrate our robust performance for one-shot animatable hand avatars creation.
Ours
wio IAtia
wil GRM
wib IMap
Ours
Wee LAtta
wio GRM
IMap
OHTA
Ours-1
Ours-1l
Ours-Ill
Ours-VV
Ours
Referesce Image
Figure 5:
Visual examples of the ablation study on the proposed components in the hand-prior learning stage (top) and the one-shot fitting stage (bottom).
4.3
Ablation Study
We conduct ablations studies in each of the two stages of our framework. Particularly, we focus on the components IGSN in the first stage while the loss terms for one-shot fitting in the second stage.
4.3.1 Interaction-awant Gaussian Splatting
The evaluation is conducted in the "train/captureO" subset with 23 pose sequences for training and 1 for testing. The quantitative results are reported Table 2 Stage-One. The results reflect that each design does bring performance gains and the best performance is obtained by the full model.
Effectiveness of GRM. To validate the effectiveness our Gaussian points refinement module, we implement variant by substituting refined Gaussian points with upsampled hand mesh points (denoted as w/o GRM. in Table 2 To further verify the effectiveness of our points modification based on the points validation prediction, implement variant by replacing the Gaussian points

---

Table 2: Ablation study on the components in our two-stage framework.
Stage-One
Stage-Two
Method
PSNR1 SSIM, LPIPS..
Method
IMap
Calib
MBias
PSNR1 SSIM
LPIPS.
OHTA*
25.93
0.880
0.156
w/o IMap.
27.76
0,900
0.139
Ours-I
AVA
26.32
0.889
0.150
w/o IAttn.
27.52
0.898
0.143
Ours-1l
IV
26.50
0.888
0,141
w/o GRM.
27.24
0.893
0.177
Ours-III
A
26.50
0.887
0.139
Ours
28.11
0.902
0.130
Ours-IV
V
26.14
0.883
0.145
Ours
V
26.56
0.890
0.133
refinement module with simply predicting offset for each point (denoted as w/o GPM. in Figure 5
The results indicate that the points modification eliminates the redundant points and densifies the detailed area based on point features, producing better rendering results.
GT
Ours
Inteructions
GT
Ours
Interactions
GT
Ours
Interactions
Figure 6: Visualization of intra- and inter-hand interactions detected by the proposed method. Areas with interactions from sparse to dense are labeled in colors from blue to red, respectively.
Effectiveness of LAttn. Figure demonstrates the predictions of the proposed interaction detection AAttn We can observe that our designed interaction detection effectively recognizes the interaction areas, including one-hand self-interaction and the interaction between two hands.
To validate the effectiveness of the interaction-aware attention module, remove it from full model (denoted as w/o LAttn. in Table P and Figure The results show that the proposed module improves the reconstruction quality of geometry deformation and texture details caused by interaction based on effective interaction detection.
Effectiveness of Identity Map. We also evaluate the effectiveness of the identity map by substi- tuting it with the identity code (vector) as in [18] (denoted as w/o IMap. in Table Figure(5) The results demonstrate that the Identity Map contributes high-fidelity hand reconstruction by capturing the fine- fine-grained texture of the target identity.
4.3.2 One-shot Fitting
We show the effectiveness of our strategies for one-shot reconstruction in Stagle and F When the identity map is replaced with identity code (IV), the performance drops indicating that our identity map can capture more accurate and fine-grained characteristics of the target hand. When there is no neural texture map bias (III), the details of the target hands are missing. Without color calibration (II), the results become worse due the color bias. When omitting the identity inversion (I), sigmificant drop in LPIPS is observed as the target identity information is not captured. Overall, we validate the effectiveness of our design for one-shot reconstruction.
Conclusion
We tackle the challenging single-image interacting hand avatar reconstruction task via an interaction- aware Gaussian splatting framework in this paper. Our framework disentangles 3D hand represen- tations into learning- based features that can be extracted by the trained network and identity maps

---

that are one-shot optimized on the hands of new subjects. Additionally, our framework employs an interaction- aware attention module and self-adaptive refinement module detect and handle regions with intra- and inter-hand interactions. The proposed method outperforms cutting-edge methods on the Interhand2.66 dataset and creates high-quality avatars for various tasks successfully.
Acknowledgments
This work was supported in part by National Key R&D Program of China (2022YFA National Science Foundation of China Grant No. 62176035, No. 62372482, No. 62476293, and No. 61936002, National Science and Technology Major Project (2000AAA0109704) Guangdong Outstanding Youth Fund (Grant No. 2021B1515020061 Shenzhen Science and Technology Program (Grant No. GJHZ20220913 122600001))) Nansha Key RD Program under Grant No.2022ZD014.
References
[1] Javier Romero, Dimitrios Tzionas, and Michael J. Black. Embodied hands: modeling and capturing hands and bodies together. ACM Transactions on Graphics, 36(6), nov 2017.
[2] Neng Qian, Jiayi Wang, Franziska Mueller, Florian Bernard, Vladislav Golyanik, and Christian Theobalt. Html: parametric hand texture model for 3d hand reconstruction and personalization. Proceedings of the European Conference on Computer Vision, pages 54-71. Springer, 2020.
[3] Korrawe Karunratanakul, Sergey Prokudin, Otmar Hilliges, and Siyu Tang. Harp: Personalized hand reconstruction from monocular rgb video. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 12802 12813, 2023.
[4] Yuwei Li, Longwen Zhang, Zesong Qiu, Yingwenqi Jiang, Nianyi Li, Yuexin Ma, Yuyao Zhang, Lan Xu, and Jingyi Nimble: non-rigid hand model with bones and muscles. ACM Transactions on Graphics. 41(4): 2022.
[5] Rolandos Alexandros Potamias, Stylianos Ploumpis, Stylianos Moschoglou, Vasileios Triantafyllou, and Stefanos Zafeiriou. Handy: Towards high fidelity 3d hand shape and appearance model. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 4670 4680, 2023.
[6] Yujin Chen, Zhigang Tu, Di Kang, Linchao Bao, Ying Zhang, Xuefei Zhe, Ruizhi Chen, and Junsong Yuan. Model-based 3d hand reconstruction via sule-supervised learning. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 10451-10460, 2021.
(7) Zheheng Jiang, Hossein Rahmani, Sue Black, and Bryan M Williams. A prohabilistic attention model with occlusion-aware texture regression for 3d hand reconstruction from single rgh image. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition, pages 758-7677 2023.
Sergey Prokudin, Michael Black, and Javier Romero. Smplpix: Neural avatars from 3d human models. In Proceedings of IEEE Winter Conference on Applications of Computer Vision, pages 1810-1819, 2021.
[9] Enric Corona, Tomas Hodan, Minh Vo, Francese Moreno-Noguer, Chris Sweeney, Richard Newcomhe, and Lingni Ma. Lisa: Learning implicit shape and appearance of hands. In Proceedings of the IEEF Conference on Computer Vision and Pattern Recognition, pages 20533-20543, 2022.
[10] Zhiyang Guo, Wengang Zhou, Min Wang, Li Li, and Houqiang Li. Handnerf: Neural radiance fields for animatable interacting hands. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 21078 -20077 2023.
[11] Xingyu Chen, Bаoyuan Wang, and Heung- Yeung Shum. Hand avatar: Free- Free-pose hand animation and rendering from monocular video. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8683-8693, 2023.
[12] Akshay Mundra, Jiayi Wang, Marc Habermann, Christian Theobalt, Mohamed Elgharib, al. Livehand: Real-time and photorealistic neural hand rendering. In Proceedings of the IEEE International Conference on Computer Vision, pages 18035 1005-1 2023.
[13] Xuan Huang, Hanhui Li, Zejun Yang. Zhisheng Wang. and Xiaodan Liang. 3d visibility- aware generalizable neural radiance fields for interacting hands. Proceedings of the AAAI Conference on Artificial Intelligence. 2024.
10

---

[14] Youngjoong Kwon, Dahun Kim, Duygu Ceylan, and Heary Fuchs. Neural human performer: Learning generalizable radiance fields for human performance rendering. Advances in Neural Information Processing Systems, 0124441-227752 2021.
[15] Marko Mihajlovic, Aayush Bansal, Michael Zollhoefer, Siyu Tang, and Shunsuke Saito. Keypointnerf: Generalizing image-based volumetric avatars using relative spatial encoding keypoints. In Proceedings of the European Conference on Computer Vision, pages 179-197, 2022.
[16] Shoukang Hu, Fangzhou Hong. Liang Pan, Haiyi Mci, Lei Yang. and Ziwei Liu. Sherf: Generalizable human nerf from single image. In Proceedings of the IEEE International Conference on Computer Vision, pages 9352-9364, 2023.
[17] Kyungmin Jo, Gyumin Shim, Sanghun Jung, Soyoung Yang, and Jaegul Choo. Cg-nerf: Conditional generative neural radiance fields for 3d-aware image synthesis. In Proceedings of the IEEE Winter Conference on Applications of Computer Vision, pages 724-733, 2023.
[18] Xiaozheng Zheng, Chao Wen, Zhuo Su, Zeran Xu, Zhaohu Li, Yang Zhao, and Zhou Xue. Ohta: One-shot hand avatar via data-driven implicit priors. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2024.
[19] Gycongsik Moon, Shoou-I Yu, He Wen, Takaaki Shiratori, and Kyoung Mu Lec. Interhand2.6mn A dataset and baseline for 3d interacting hand pose estimation from a single rgb image. In Proceedings of the European Conference on Computer Vision, pages 548-564, 2020.
[20] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. Smpl: A skinned multi-person linear model. ACM Transactions on Graphics, 34(6):1-16, 2015.
[21] Thiemo Alldieck, Marcus Magnor, Bharat Lal Bhatnagar, Christian Theobalt, and Gerard Pons-Moll. Learning to reconstruct people in clothing from single rgb camera. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1175 1186, 2019.
[22] Verica Lazova, Eldar Insafutdinov, and Gerard Pons-Mol 360-degrce textures of people in clothing from single image. 2019 International Conference on Vision (3DV), pages 653-653. IEEE, 2019.
[23] Xiangyu and Chen Change Loy. 3d human texture estimation from a single image with transformers. In Proceedings of the IEFE International Conference on Computer Vision, pages 13849-13855, 2021.
[24] Badour AlBahar, Shunsuke Saito, Hung- Tseng, Changil Kim, Johannes Kopf, and Jia-Bin Huang. Single-image 3d human digitization with shape-guided difflusion. In SIGGRAPH Asia 2023 Conference Papers, pages 1-11, 2023.
[25] David Svitov, Dmitrii Gudkov, Renat Bashirov, and Victor Lempitsky. Dinar: Diffusion inpainting of neural textures for one-shot human avatars. In Proceedings of the IEFE International Conference on Computer Vision, pages 7062-7072, 2023.
[26] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
[27] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelnerf: Neural radiance fields from one or few images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 47778-4577.1 2021.
[28] Jianchuan Chen, Wentao Yi, Liqian Ma, Xu Jia, and Huchuan Lu. Gm-nerf: Learning generalizable model-based neural radiance fields from view images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2023.
[29] Fangzhou Hong, Zhaoxi Chen, Yushi LAN, Liang Pan, and Ziwei Liu. EVA3d: Compositional 3d human generation from 2d image collections. In International Conference Learning Representations, 2023.
[30] Qiangian Wang. Zhicheng Wang. Kyle Genova, Pratul P Srinivasan, Howard Zhou, Jonathan T Barron, Ricardo Martin-Brualla, Noah Snavely, and Thomas Funkhouser. Ibrnet: Learning multi-view image-bascd rendering. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pages 4690-4699,20 2021.
[31] Haotong Lin, Sida Peng. Zhen Xu, Yunzhi Yan, Qing Shuai, Hujun Bao, and Xiaowei Zhou. Efficient neural radiance fields for interactive free-vien vie-vit video. In SIGGRAPH Asia Conference Proceedings. 2022.
11

---

[32] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4)):--1, 2023.
[33] Wojeiech Zielonka, Timur Bagautdinov, Shunsuke Saito, Michael Zollhöfer, Justus Thies, and Javier Romero. Drivable 3d gaussian avatars, arXiv preprini arXiv:2311.0885 2023.
[34] Zheheng Jiang, Hossein Rahmani, Sue Black, and Bryan M Williams. 3d points splatting for real-time dynamic hand reconstruction. arXiv preprint urXiv:2312.137700 2023.
[35] Zhiyin Qian, Shaofei Wang, Marko Mihajlovie, Andreas Geiger, and Siyu Tang. 3dgs-avatar: Animatable avatars via deformable 3d gaussian splatting. Proceedings of the IEEE Conference Computer Vision and Puttern Recognition, 2024.
[36] Shoukang Hu and Ziwei Liu. Gauhuman: Articulated gaussian splatting from monocular human videos. arXiv preprint arXiv:2312.02975. 2023.
[37] Marcel C Bühler, Kripasindhu Sarkar, Tanmay Shah, Gengyan Li, Daoye Wang, Leonhard Helminger, Sergio Orts-Escolano, Dmitry Lagun, Oumar Hilliges, Thabo Beeler, et al. Preface: A data-driven volumetric prior for few-shot ultra high-resolution face synthesis. In Proceedings of the IEEE International Conference on Computer Vision, pages 3402-3413, 2023.
[38] Charles R Qi. Hao Su. Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 052-660,22 2017.
[39] Songyou Peng, Michael Niemeyer, Lars Mescheder, Marc Pollefeys, and Andreas Geiger. Convolutional occupancy networks. In Proceedings of the European Conference Computer Vision, pages Springer. 2020.
[40] Zi-Xin Zou, Zhipeng Yu, Kuan-Chen Guo, Yangguang Li, Ding Liang, Yan-Pei Cao, and Song- Hai Zhang. Triplane meets gaussian splatting: Fast and generalizable singlc-view 3d reconstruction with transformers. arXiv preprint arXiv:2312.09144 2023.
[41] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses real-time style transfer and super-resolution. In Proceedings of the European Conference on Computer Vision, pages 694-711, 2016.
[42] Mengcheng Li. Liang An. Hongwen Zhang, Lianpeng Wu, Feng Chen, Tao Yu, and Yebin Liu. Interacting attention graph for single image two- woo-aad reconstruction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2761-2770, 2022.
[43] Zhengdi Yu, Shaoli Huang, Chen Fang, Toby P Breekon, and Jue Wang. Acr: Attention collaboration based regressor for arbitrary two-hand reconstruction. In Proceedings of the IEEE Conference on Computer Vision and Pottern Recognition, pages 12955 12964, 2023.
[44] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980, 2014.
[45] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features a perceptual metric. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 586-595, 2018.
[46] Umme Sara, Morium Akter, and Mohammad Shorif Uddin. Image quality assessment through fsim, ssim, mse and psnr a comparative study. Journal of Computer and Communications, 7(3):8 18, 2019.
[47] Zhou Wang. Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600-61 612,, 2004.
[48] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to- text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836 3847, 2023.
[49] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tote Xiao, Spencer Whitehead, Alexander C, Berg, Wan- Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. arXiv:2304.088 026443 2023.
12

---

A
Appendix
A.1 Application.
As shown in Figure 7 We demonstrate more visual results of various applications, including in-the-wild results from real-captured images, text-to-avatar, and texture editing. For text-to-avatar, utilize ControlNet 44 with depth maps as condition information for hand image generation. For in-the-wild results, we use ACR to estimate hand pose and camera parameters from real-captured images. These results clearly show that our method can be applied to in-the-wild images and obtain considerable results.
"hand, spifer man""
"hand. Mur"
"hand
In-hh-willd
Editing
Figure 7: Visual examples of the proposed method in various scenarios, including text-to-avatar, in-the-wild reconstruction from real images, and texture editing.
A.2
Statistical Significance
To evaluate the statistical significance, we run the proposed method five times with different random initiation of identity map in the second stage. Table presents the mean and standard deviation of the results.
Table Mean and standard deviation of the performance of the proposed method.
Method
PSNRT
SSIMT
LPIPSJ
Ours
46.6027+0.0233
0.8879 0.0099-000009
0.1342--00000
A.3 Visualization of GRM
Figure presents qualitative comparisons of hand geometry rendering results between our Gaussian refinement modules and vanilla GS with mesh upsampling. We can see that, the hand points initialized from hand parameters can only provide approximate hand geometry while refined Gaussian points excel in producing detailed texture and realistic hand geometry with the help of the Gaussian points Refinement module.
13

---

MeshUpp GRM
MpshUp
GRM
GT
MeshUp GRM
MeshUp
GRM
GT
Figure 8: Qualitative comparisons between Gaussians by mesh upsampling (denoted as MeshUp) and the proposed GRM.
A.4 Data Selection and Preprocessing
We use the same captures for training with OHTA [8] in stage one and save the pose sequence '0275 babybird' for testing. For the evaluation of stage two, we conduct experiments on 'test/CaptureO" including four pose sequences 'ROMO1_No _n___ntrractton_ 2_riion_2____ 2_Hand', 'ROMO1 No Interaction 2
'0275_left- Jeft-
__anidd_____,d'', Hand',
[.OM09_Ineeaactiin Lot___ttraatio_Fnneeeeee Fingers_Touchhnnnee
and 'ROM09 Interaction Fingerss__________ Taaccrs_Tuuchng__2
For image preprocessing, crop the hand region with the bounding boxes and resize the cropped area 256 256 consistent with previous methods [3 [8]. adopt SAM 49 produce hand masks for better
segmentation compared to MANO-rendered masks.
A.5
More Ablations
Figure demonstrates visual examples shadow disentanglement (top) qualitative results of four ablation studies (bottom). The quantitative results of the four ablation studies are provided in Table
Image
Alhrda
Image
Albrdo
Image
Albedo
Shadow
Ablation Hand Nuas.
Ablation Mask
Rendering
Ablation S1 mis Cam..
Ablation Mesh
CT
Ours
Studow Low C. Non.
Reforeace
Ours
w/Noby Mesh
Neies
Figure Visual examples of shadow disentanglement (top) and four ablation studies (bottom).
Shadow Disentanglement (denoted as Shadow). Figure 9 (top) demonstrates visual examples shadow disentanglement including shaded image, an albedo image, and shadow image for each pair of hands. This figure clearly shows the shadow (dark) areas, which suggests that our method can disentangle albedo and shadow.
Ablation study single-hand images (denoted Ablation Hand Num.). The advantage of using images of both hands is that two-hand images provide complementary information compared with single-hand images, which helps improve the model's performance. We experiment to compare the performance between single-hand hand images and two-hand images by masking one of the hands from the reference image. The results show that our method achieves better performance compared with the single-hame hand baseline, which is reasonable two hand images contain complementary information.
14

---

Ablation study segmentation method (denoted as Ablation Mask), To validate the impact of segmentation masks, consider masks predicted by SAM and the ground-truth meshes. We observe that SAM masks enhance the performance of our model, since they are better aligned with hands and prevent background clutters.
Ablation S1. Ablation S1 shows the visual results three different ablation settings: including (i) the of camera parameters, which is a variant of our method without the camera parameters (w/o Cam.) and suffers from obvious performance drop: (ii) adding shadow coefficient, which incorporates shadow coefficient prediction into the proposed method (w/ Shadow); ) (iii) Low Gaussian Points, which reduces the number of Gaussian points to 24k show that coarsen hand boundaries caused by fewer numbers of Gaussian points.
Ablation study mesh estimation method (denoted Ablation Noise). We hand parameters estimated by ACR 4 to analyze how the accuracy mesh reconstruction impacts the final results. The performance of our method with the predicted parameters still satisfying. which suggests our method is robust to noise hand meshes to certain extent.
Table 4: The quantitative results of four ablation studies.
Ablation Hand Num.
Ablation Noise
Method
PSNRT
SSIM
LPIPS.
Method
Mesh
PSNR1
SSIM
LPIPSI
Ours
26.14
0.869
0.161
Ours Ours
26.14 25.83
0.869
0.161
Ours- Rught
24.81
0.852
0.178
ACR
0.864
0.167
Ours-Left
25.58
0.858
0.172
w/o Cam. in Ablaiton S1
Ablation Mask
Method
PSNRT
SSIM
LPIPS.
Method
Mask
PSNRT
SSIM
LPIPS|
Ours
28.11 25.91
0.902 0.862
0.130
Ours
SAM
26.52 25.99
0.888
0.135
w/o Cam.
0.198
Ours
Mesh
0.877
0.142
A.6 Implementation of OHTA*
Since OHTA [8] as single-hand reconstruction method canmot be directly applied to our scenario, we instead use our pre-trained model and only compare the design of the fine-tuming stage. We implement OHTA* with the texture inversion stage and texture fitting stage as described in its paper. In the texture inversion, we keep the network weights frozen and optimize the identity code along with per-channel color calibration coefficients to produce similar appearance texture feature MI.P for texture feature extraction constrain the texture-fitting results some reference views to he close to the rendering results hefore texture-fitting.
the target identity of the input image. In the texture fitting, we fine-tune the
A.7 Limitations and Society Impact
Although we have greatly shortened the tne-tuning process, per-identity training still limits the application compared to single forward inference. Moreover, separate optimization for each identity hinders the model from integrating similar identity information optimized in the fine-tuning stage.
When facing serious errors in pose estimation, our method may fail to model the proper appearance through the wrong alignment caused by misleading hand parameters shown in Figure [T0) Severe estimation errors inevitahly cause degraded performance, which is also mentioned in the paper of OHTA [T].
Our method has positive impact society it facilitate sign language production by creating high-fidelity, animatable hand avatars with interaction. There are no negative societal impacts concerning our work.
Reference
Keypsinis
Mesh
Mask
Masked Image
Rendering
Reference
Keypeints
Mesh
Mask
Masked imnge
Rendering
Figure 10: Visual examples of failure cases.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The paper's 's contributions and scope are accurately stated in abstract the end of the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be pereeived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings. It is fine to include aspirational goals motivation long it is clear that these goals attained by the paper.
2. Limitations
Question: Does paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The discussion of limitations is involved the appendix (Sec. 6
Guidelines:
The answer NA means that paper has no limitation while answer means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust results are violations these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested a few datasets with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
grounds for rejection,
Theory Assumptions and Proofs
Question: For each thcoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The paper does not include theoretical results formal proofs.
Guidelincs:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
16

---

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by
formal proofs provided in appendix or supplemental material.
Theorems and Lemmas the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
The setup of experiments is provided Setup (Sec. 4.1 for reproduction.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others replicate the model with the same dataset, or provide access the model. In general. releasing code and data often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (c.g., in the case of a large language model), releasing of model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g.. a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g.. with an open-source dataset or instructions for how to construct
dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome to describe the particular way they provide for reproducibility. In the of cosed-source models, it may be that access to the model is limited some way (e.g.. to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open faithfully reproduce the main experimental results, described supplemental material?
the data and code, with sufficient instructions
Answer: [Yes]
Justification:
Code and models will be released upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/bblicc/
quides/CodeSubmn sei.bboiiion for more details.
While we encourage the release of code and data, we understand that this might not be possihle, so "Na" an acceptable answer. Papers cannot rejected simply for not including code, unless this is central to the contribution (e.g.. for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: :/ :/nniss//.///////////e:::eee cc/public/ qu..dess/ooeeSuubllesssoool..... for more details.
17

---

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) recommenedd, but including URLs to data and code is permitted.
6. Experimental Setting/Detan Details
Question: Does the paper specify all the training and test details (c.g,, data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The training and test details are provided in Sctup (Sec. 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate informa- tion about the statistical significance of the experiments?
Answer: [Yes]
Justification:
The error bars are reported in the appendix (Sec. 6
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should given (e-g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error than state that they not verified.
CL, if the hypothesis of Normality of errors
asymmetric distributions, authors should be careful not show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). crror bars are reported tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question: For each experiment, docs the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification:
The information computer resources is provided in Setup (Sec. 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
18

---

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer:
[Yes]
Justification:
The research conducted in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer means that authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., , if there is a special consideration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts the work performed?
Answer: [Yes]
Justification: The paper discuss both potential positive societal impacts and negative societal impacts of the work performed in the appendix (Sec.
Guidelines:
The answer NA means that there is societal impact of the work performed.
the authors answer NA No. they should explain why their work has societal impact why the paper does address societal impact.
Examples of negative socictal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point out. For example, the quality generative models could used to generate deepfakes for disinformation. On other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
legitimate to point out that an improvement in
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that been put in place responsible release data or models that have a high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk misuse or dual-use should he released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
19

---

We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are creators or original owners assets (e.g., code, data, models), used the paper, properly credited and are license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The creators or original owners of assets used in the paper are properly credited and the license and terms of use are explicitly mentioned in the Setup (Sec. 4.1
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are relcased, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license a dataset.
For existing datasets that re-packaged. both the original license and the license of the derived asset (if it changed) should provided.
this information not available online, the authors encouraged reach out the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The NA means that the paper does not release
Researchers should communicate the details of the dataset/code/mmode as part of their sub- missions structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least minimum wage in country data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
20

---

Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer means that paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, approval (or equivalent) may be required for any human subjects research. If you ohtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such the institution conducting the review.
21


Reviews:
[{"review_id": "JNew59zi6t", "reviewer": "Reviewer_NUjU", "summary": "- The author extends the concept of one-shot hand avatar creation from the single hand in the previous OHTA paper to two hands.\n- The author proposes a novel two-stageGS framework for the reconstruction and rendering of avatars. This framework utilizes 2D identity maps to represent identity information and assist in the learning of neural textures. Additionally, an attention mechanism and a Gaussian refinement module are designed to handle the interaction information.\n- The author demonstrates the superior performance of their method compared to the baseline on the Interhand2.6M dataset.", "questions": "- The OHTA* method compared in the article seems more like an ablation of the full model. The performance of the OHTA algorithm reproduced by the authors is worse than in the original paper. Can an explanation be provided on how this baseline was implemented, and can some reasons for the poorer performance be analyzed?\n\n- The paper incorporates camera intrinsic and extrinsic parameters as part of the Geometric Encoding. What are the advantages of introducing this information? Is there any related ablation analysis?\n\n- Is the initial mesh of the bimanual results shown by the author estimated or taken from the dataset's ground truth? If it is estimated, what estimation method was used? How is the error in pose estimation addressed?", "limitations": "The author has already addressed the limitations of the method.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The author has successfully applied the Gaussian splatting module for the first time in the problem of one-shot hand avatar creation.\n\n- The method proposed in the paper has shown improved performance on the Interhand dataset.\n\n- The paper is easy to follow.", "weaknesses": "- The article represents an incremental work.\n\n- Initially, it is not convincing why one would construct a hand avatar using images of both hands with interaction instead of building them separately. What are the advantages?\n\n- Secondly, the advantage of the one-shot method lies in addressing in-the-wild scenarios. The pioneering work in the single hand direction, OHTA, demonstrated a multitude of in-the-wild results, but this paper does not. This raises doubts about the effectiveness of this method in in-the-wild scenarios.\n\n- The method's modeling of the hand is not sufficiently detailed. Previous methods such as Handavatar, OHTA, etc., have considered shadows caused by occlusions, but it is not clear how this article's method learns such priors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer’s valuable comments and suggestions. We have conducted extensive ablation studies and provide more in-the-wild results to further verify the proposed method. Our responses are listed as follows: \n\n**W1 Contribution and novelty**: We feel it is necessary to clarify the technical contributions of the proposed method from the following perspectives:\n\nFirst, constructing two-hand avatars via simple extensions of one-hand baselines faces certain limitations. Particularly, it is challenging for single-hand methods to handle severe occlusions and inter-hand interference, which results in inferior performance. This can be validated by our new quantitative (the table in our following response to W2) and qualitative comparisons (RF-Fig. 2, Ablation Hand Num.) provided in the rebuttal. Moreover, single-hand methods do not include modules to leverage the structure and texture similarity of hands, which are informative in various scenarios. Therefore, extensive methods (VANeRF[1], ACR[2], and Im2hands[3]) are tailored specially for the two-hand task.\n\nSecond, three novel components have been introduced in our method, including (i) the disentangled 3D presentation to improve flexibility and exploit data priors, (ii) the interaction-aware attention module to handle complex interactions, and (iii) a Gaussian refinement module to improve rendered image quality. The effectiveness of these modules has been validated by the ablation studies in our paper.\n\nLast but not least, we are glad that all other reviewers found the proposed method novel and effective, such as ``novel and intuitive\" (Reviewer #gWjw), technically sound (Reviewer #eWJx and #mjNr), and one of the first works\" (Reviewer #eWJx and #mjNr). We believe such appreciation can indicate that the proposed method is not incremental.\n\nHence, we sincerely hope the novel and contributions of our method can be reconsidered. We are open to further discussion and will try our best efforts to address your concern.\n\n[1]Xuan Huang et al., 3d visibility-aware generalizable neural radiance fields for interacting hands. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 2400–2408, 2024.\n\n[2]Zhengdi Yu et al., Acr: Attention collaboration-based regressor for arbitrary two-hand reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12955–12964, 2023.\n\n[3]Jihyun Lee et al., Im2hands: Learning attentive implicit representation of interacting two-hand shapes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21169–21178, 2023.\n\n**W2 The advantages of using images of both hands**: The advantage of using two-hand images is that they provide more complementary information compared with those of single hands, especially when severe inter-hand occlusions occur. To validate this, we compare the performance of our method with that of the single-hand counterpart by masking one hand in reference images. The quantitative results are listed as follows:\n\n|   Method   |   PSNR    |   SSIM    |   LPIPS   |\n| :--------: | :-------: | :-------: | :-------: |\n|    Ours    | **26.14** | **0.869** | **0.161** |\n| Ours-Right |   24.81   |   0.852   |   0.178   |\n| Ours-Left  |   25.58   |   0.858   |   0.172   |\n\nThe above results show that our method with two-hand images outperforms that with single-hand images significantly. Moreover, RF-Fig. 2 Ablation Hand Num. demonstrates the visual examples of this experiment. These examples also suggest that using both hands increases the quality of rendered images.\n\nCompared to single-hand modeling, the intra- and inter-hand interaction exacerbates information loss and introduces complex geometric deformations. It is necessary to study the interacting hand reconstruction. Extensive efforts have been made by related research communities to reconstruct interacting hands from a single image, such as VANeRF, ACR, and IntagHand. To improve the performance of interacting hand reconstruction, we propose an interaction-aware attention module and a self-adaptive Gaussian refinement module. These modules enhance image rendering quality in areas with intra- and inter-hand interactions.\n\nFurthermore, our method can construct a hand avatar using images of both hands with interaction or single-hand images. Since there are situations in which we only have single-hand images or interacting hand images, our method has a wider application than single-hand reconstruction methods. \n\n**W3 In-the-wild results**: As shown in RF-Fig. 1, We demonstrate more visual results of various applications, including in-the-wild results from real-captured images, text-to-avatar, and texture editing. For text-to-avatar, we utilize ControlNet with depth maps as condition information for hand image generation. For in-the-wild results, we use ACR to estimate hand pose and camera parameters from real-captured images. These results clearly show that our method can be applied to in-the-wild images and obtain considerable results.\n\n**W4 Shadow modeling**: Handavatar and OHTA disentangle the pose-related information and further predict the shadow coefficient so that they can visualize the shadow and albedo separately. We disentangle pose-related information from identity-related information to ensure generalization ability. Instead of predicting the shadow coefficient, we render the shadow and albedo simultaneously. However, our method is flexible in choosing shadow modeling strategies. To demonstrate this, we conduct an experiment that uses disentangled pose-related features to predict shadow coefficients as in OHTA, allowing for the separate modeling of shadows and albedos. The visual results are shown in RF-Fig. 2 Shadow. From these results, it is apparent that our method successfully separates the shadows (e.g. the area beneath the palm). The qualitative results are shown in RF-Fig. 2 w/ Shadow of Ablation S1.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mZnBqzvU90", "reviewer": "Reviewer_gWjw", "summary": "This paper proposes a novel two-stage interaction-aware GS framework to create animatable avatars for interacting hands from single-image inputs. The proposed method disentangle the 3D presentation of hands into optimization-based identity maps and learning-based latent geometric features and neural texture maps, and exploits cross-subject hand priors and refines 3D Gaussians in interacting areas. Experiments are conducted on the Interhand2.6M dataset with state-of-the-art method comparisons.", "questions": "1) In lines 85-87 of page 3, why can disentangling hand prior learning reduce the cost of one-shot fitting？Is this related to introducing texture map bias to accelerate one shot fitting\n2) Why is the interacting label $d(q)$ binarized, rather than applied by soft activation?\n3) An important baseline, OHTA, was re-implemented and compared in the experiment. Can you provide more details?", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The paper is well written and easy to follow, and it is technically sound to disentangle geometric and texture feature learning to achieve prior learning across different subjects.\n2) The idea of the Interaction-aware Attention module is novel, which detect the interation-aware geometric deformation in an intuitive way.\n3) The quantitative and qualitative results are of good quality and surpass the current SOTA on Interhand2.6M, and the ablation studies are quite extensive.\n4) Comprehensive qualitative ablation studies are shown. The authors additionally present qualitative results on the interaction areas, showing enhanced rendering quality.", "weaknesses": "1) The data flow in Figure 3 is somewhat unclear. For example, what does the dotted line from the Interaction Points to the Interaction-Aware Attention present? In addition, using different colored sub boxes for geometric flow and texture flow can help readers more intuitively understand the disentanglement of hand priors.\n2) In line 119 of page 4, there is a typo error. 'Sec.' is missing.\n3) The generalization of in-the-wild images has not been demonstrated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your acknowledgment of the technical contributions and novelty of our method. We have considered your as well as other reviewers' suggestions carefully and tried to solve them as follows: \n\n**W1&W2 Presentation**: Thank you for pointing this out. We will enhance our presentation in Figure 3 and revise the typo accordingly. The dotted line from the Interaction Points to the Interaction-Aware Attention means that we feed the information of interaction points into the Interaction-Aware Attention module. Considering the page limitation, the revised version of Figure 3 using different colored sub-boxes for geometric flow and texture flow will be presented in the future version.\n\n**W3 In-the-wild results**: RF-Fig. 1 shows more real-world results and applications of our proposed method, affirming its effectiveness in real-world scenarios and its suitability for handling out-of-distribution data. RF-Fig. 1 comprises three challenging tasks, i.e., text-to-avatar, real image in-the-wild reconstruction, and texture editing. The proposed method handles these tasks effectively. \n\n**Q2 The cost of one-shot fitting**: Yes, introducing texture map bias helps accelerate one-shot fitting due to the reduction of learnable parameters. Compared to directly fine-tuning the network (427 M), optimizing the texture map bias (0.65M) reduces the fine-tuning time significantly.\n\n**Q2 Interacting label**: We adopt binarized interacting labels instead of utilizing probabilistic ones because we propose a simple and effective strategy to detect interacting points (P6, L193-197). Specifically, we calculate the difference between the neighboring point sets of posed hand meshes and a canonical mesh. If the difference is above the user-defined threshold, the corresponding interacting label is set to 1 otherwise 0. In our experiments, we find this strategy effective enough to detect self-interacting and cross-interacting points.\n\n**Q3 Implementation of OHTA**: Because the code of OTHA is not published before the deadline for paper submission and the pre-trained model of OTHA cannot be used in our scenario, we utilize our own pre-trained model and only compare the fine-tuning stage design. Specifically, we implement OHTA* with the texture inversion stage and texture fitting stage following its paper. In the texture-inversion stage, we keep the weights of the pre-trained network frozen and optimize the identity code along with per-channel color calibration coefficients to fit the input image. In the texture-fitting stage, we fine-tune the MLP for texture feature extraction and impose the constraint that the texture-fitting results of reference views should be close to the rendering results before texture-fitting.\n\nConcerning the inferior performance of OHTA*, we consider that the previous number of training steps might be insufficient for OHTA* to fully fit the reference image. To investigate this, we further prolong the training steps (denoted as OHTA**) and obtain better performance as follows:\n\n| Method |   PSNR    |   SSIM    |   LPIPS   | Fine Tuning Time |\n| :----: | :-------: | :-------: | :-------: | :--------------: |\n|  Ours  | **26.14** | **0.869** |   0.161   |   2.5 minutes    |\n| OHTA*  |   25.31   |   0.851   |   0.184   |    5 minutes     |\n| OHTA** |   25.96   |   0.864   | **0.160** |    25 minutes    |\n\nYet, our method still achieves a better balance between rendering quality and fine-tuning time.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tQdkogENhC", "reviewer": "Reviewer_eWJx", "summary": "The paper proposes an approach to achieve one-shot interacting hand avatar reconstruction via 3D Gaussian Splatting. The authors design a two-stage framework: the first stage learns learning-based features and optimization-based identity maps, and the second stage performs one-shot reconstruction by optimizing only the identity map. To better capture the relationship between the two hands, they devise an interaction-aware attention module. Moreover, they design a self-adaptive GS density control approach for this task. The experiments demonstrate the effectiveness of their designs and show that the approach achieves state-of-the-art performance.", "questions": "1. Why should we use interacting hands for avatar creation? What are the benefits compared to single-hand modeling?\n2. What is the detailed implementation of OHTA? The results appear to be worse than the reported results.\n3. How does the accuracy of mesh reconstruction impact the final results?", "limitations": "The authors present the \"Limitations and Society Impact\" section.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper presents the first framework capable of achieving one-shot animatable two-hand avatars.\n2. The disentangled designs for one-shot reconstruction, using optimization-based identity maps for the one-shot stage, are technically sound.", "weaknesses": "1. For interacting hands, significant shadows occur between the two hands. The proposed method does not account for shadow modeling, which overlooks the dynamic nature of the hands and makes the approach less practical for real-world scenarios to some extent.\n2. The experimental results are not comprehensive.\n- It would be beneficial to present some in-the-wild testing results to demonstrate the method's plausibility for real applications. This would also justify the design for addressing out-of-distribution (OOD) data.\n- Including some failure cases would help to illustrate the boundaries of the work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your suggestive comments. We have performed multiple experiments and presented additional real-world results to further validate the proposed method. Our point-by-point responses are listed as follows: \n\n**W1 Shadow modeling**: Thank you for your suggestive advice. Our current framework indeed focuses on the disentanglement of poses and identities and ignores shadow modeling. However, as we do not impose any constraint on shadows, our framework is of great flexibility and hence is easy to incorporate existing shadow models. To demonstrate this, we follow OHTA to disentangle pose-related features and predict shadow coefficients, which allows our method to model shadows and albedos separately. As shown in RF-Fig. 2, our method with this enhancement successfully separates shadows caused by occlusions. Moreover, we also provide more in-the-wild examples regarding three challenging tasks in RF-Fig. 1. These examples indicate that even our current model can handle real-world scenarios. From these two aspects, we believe the proposed method is a feasible solution for various applications.\n\n**W2 In-the-wild results**: Thank you for your advice. As shown in RF-Fig. 1, we demonstrate more visual examples of three scenarios, including in-the-wild reconstruction, text-to-avatar, and texture editing. In the in-the-wild results, we use ACR to estimate hand pose and camera parameters from real-captured images. These results clearly show that our method can be applied to in-the-wild images and obtain considerable results. These results will be added to our revised paper.\n\n**W3 Failure cases**: We add the demonstration of failure cases as shown in RF-Fig. 3. If severe hand mesh estimation errors occur, our method may fail to model the proper appearance. We notice previous methods also suffer from this issue (as mentioned in the OHTA and VANeRF paper), and a better mesh recovery method is necessary in this case. \n\n**Q1 The benefits of using interacting hands for avatar creation**: The advantage of using two-hand images is that they provide more complementary information compared with single hands, especially when severe inter-hand occlusions occur. To validate this, we compare the performance of our method with that of the single-hand counterpart by masking one hand in reference images. The quantitative results are listed as follows:\n\n|   Method   |   PSNR    |   SSIM    |   LPIPS   |\n| :--------: | :-------: | :-------: | :-------: |\n|    Ours    | **26.14** | **0.869** | **0.161** |\n| Ours-Right |   24.81   |   0.852   |   0.178   |\n| Ours-Left  |   25.58   |   0.858   |   0.172   |\n\nThe above results show that our method with two-hand images outperforms that with single-hand images significantly. Moreover, RF-Fig. 2 Ablation Hand Num. demonstrates the visual examples of this experiment. These examples also suggest that using both hands increases the quality of rendered images.\n\nCompared to single-hand modeling, the intra- and inter-hand interaction exacerbates information loss and introduces complex geometric deformations. It is necessary to study the interacting hand reconstruction. Extensive efforts have been made by related research communities to reconstruct interacting hands from a single image, such as VANeRF, ACR, and IntagHand. To improve the performance of interacting hand reconstruction, we propose an interaction-aware attention module and a self-adaptive Gaussian refinement module. These modules enhance image rendering quality in areas with intra- and inter-hand interactions.\n\nFurthermore, our method can construct a hand avatar using images of both hands with interaction or single-hand images. Since there are situations in which we only have single-hand images or interacting hand images, our method has a wider application than single-hand reconstruction methods. \n\n**Q2 Implementation of OHTA***: Please note that OHTA is a single-hand reconstruction approach, and **its setup and dataset completely differ from ours**. Because its pre-trained model cannot be used in our scenario, we utilize our own pre-trained model and only compare the fine-tuning stage design. The code of OTHA is not published before the deadline for paper submission, hence we implement OHTA* with the texture inversion stage and texture fitting stage following its paper. In the texture-inversion stage, we keep the weights of the pre-trained network frozen and optimize the identity code along with per-channel color calibration coefficients to fit the input image. In the texture-fitting stage, we fine-tune the MLP for texture feature extraction and impose the constraint that the texture-fitting results of reference views should be close to the rendering results before texture-fitting.\n\nConcerning the inferior performance of OHTA*, we consider that the previous number of training steps might be insufficient for OHTA* to fully fit reference images. To investigate this, we further prolong the training steps (denoted as OHTA**) and obtain better performance as follows:\n\n| Method |   PSNR    |   SSIM    |   LPIPS   | Fine Tuning Time |\n| :----: | :-------: | :-------: | :-------: | :--------------: |\n|  Ours  | **26.14** | **0.869** |   0.161   |   2.5 minutes    |\n| OHTA*  |   25.31   |   0.851   |   0.184   |    5 minutes     |\n| OHTA** |   25.96   |   0.864   | **0.160** |    25 minutes    |\n\nYet, our method still achieves a better balance between rendering quality and fine-tuning time.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0GZU6zioRE", "reviewer": "Reviewer_mjNr", "summary": "This paper proposes a method to create a 3D Gaussian Splatting (GS)-based avatar for interacting hands from single-image inputs. Its main contribution is to propose a two-stage GS framework (1) to leverage cross-identity priors via learning-based features and also to (2) well preserve per-identity information via optimized identity maps. Additionally, it proposes an interaction-aware attention module and a Gaussian refinement module to further exploit the hand interaction context for more realistic rendering. In the experiments, the proposed methods outperforms the existing methods based on generalizable and one-shot NeRFs.", "questions": "(1) In lines 94-95, the paper claims that the rendering quality of the existing MANO-based methods “is constrained by the coarse geometry and sparsity of the MANO mesh”. However, to my knowledge, HARP [3] and Handy [5] use higher resolution-version of MANO meshes to already address this issue. What would be the advantage of using GS compared to using such higher-resolution hand meshes?\n\n(2) In lines 118-119, the paper states that the proposed method \"reduce[s] the time consumption of one-shot avatar reconstruction\". Have you tried comparing the reconstruction time of the proposed method against OHTA [18]?", "limitations": "The authors discussed the limitations in Sec. A.4 in Appendix.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "**(1) Good presentation.** Overall, the paper is well-organized and technical explanations are sound. The figures are also clearly presented.\n\n**(2) Sufficient technical novelty.** The disentanglement of hand representation into learning-based features and optimization-based identity maps sounds reasonable. Also, to the best of my knowledge, this is one of the first works on generating avatars for interacting hands from single image inputs.\n\n**(3) Good experimental validation.** The paper reports sufficient ablation study results and performs adequate comparisons with the existing baselines.", "weaknesses": "**(1) Effects of off-the-shelf MANO regressor and mask detector.** The proposed avatar reconstruction method (Sec. 3.3) uses the outputs of off-the-shelf MANO regressor and segmentation model. It would be informative to include the discussions on how the quality of these outputs might affect the final rendering quality.\n\n**(2) Non-smooth hand boundaries in rendering results.** In Fig. 4, I’ve noticed that the boundaries of hands rendered using the proposed method are less smooth compared to the existing methods. Does it indicate that the proposed GS-based method (implicitly) learns less smooth hand geometries than the other NeRF or mesh-based methods? More discussions regarding this would be informative.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer’s confirmation of sufficient technical novelty. To address the concerns of the reviewer, we conduct several ablation studies and provide extra visual examples in the RF. Our responses are listed as follows: \n\n**W1 Effects of off-the-shelf MANO regressor and mask detector**: Thank you for your advice. We add experiments accordingly as follows: \n\n(i) We experiment to evaluate the effects of the off-the-shelf MANO regressor (ACR). The qualitative results are shown in RF-Fig. 2 Ablaiton Mesh and the quantitative results are listed as follows:\n\n| Method | Mesh |   PSNR    |   LPIPS   |   SSIM    |\n| :----: | :--: | :-------: | :-------: | :-------: |\n|  Ours  |  GT  | **26.14** | **0.869** | **0.161** |\n|  Ours  | ACR  |   25.83   |   0.864   |   0.167   |\n\nThe performance of our methods with parameters estimated by ACR is still satisfactory. This is because we have added regulation to texture map bias to prevent our model from over-fitting the incorrect alignment between hand appearance and geometry, which addresses minor estimation errors effectively.\n\n(ii) To validate the impact of using masks produced by SAM, we conduct ablation experiments between using the SAM masks and the mesh rendering masks. The qualitative results  are shown in RF-Fig. 2 Ablaiton Mask and the quantitative results are listed as follows:\n\n| Method | Mask |   PSNR    |   SSIM    |   LPIPS   |\n| :----: | :--: | :-------: | :-------: | :-------: |\n|  Ours  | SAM  | **26.52** | **0.888** | **0.135** |\n|  Ours  | mesh |   25.99   |   0.877   |   0.142   |\n\nWe can see that using the masks produced by SAM improves the model performance, due to the better alignment with hand boundaries,  which helps to mitigate background clutters.\n\n**W2 Non-smooth hand boundaries**: Thank you for pointing this out. Compared to NeRF-based methods, GS-based methods usually use fewer rendering points, which may lead to less smooth hand boundaries. To confirm this, we further lower the number of Gaussian points to 24k in our method. The experimental results are shown in RF-Fig. 2 (denoted as Low G. Num. in Ablation S1). We find that hand boundaries indeed deteriorate due to the fewer number of Gaussian points.\n\n**Q1 The advantage of GS compared to high-resolution hand meshes**: The advantage of using GS is that GS is more flexible compared to parametric high-resolution hand meshes. The geometry of parametric high-resolution hand meshes is relatively fixed and \nthus it's hard for them to present fine-grained geometric deformation of interacting hands. To alleviate such limitation of high-resolution hand mesh, we devise a Gaussian refinement module that adaptively adjusts the number and positions of 3D Gaussian according to different hand poses and identities, which results in better hand geometry in the rendered images.\n\n**Q2 Comparing the Reconstruction Time with OHTA**: The one-shot avatar reconstruction of OHTA takes 56 minutes on an A100 GPU (according to the OHTA paper) while our method takes 2.5 minutes on an A6000 GPU. For the fairness of comparison, we conduct an experiment to test the fine-tuning cost of our method and OHTA, of which the results are listed below:\n\n| Method |   PSNR    |   SSIM    |   LPIPS   | Fine Tuning Time |\n| :----: | :-------: | :-------: | :-------: | :--------------: |\n|  Ours  | **26.14** | **0.869** |   0.161   |   2.5 minutes    |\n| OHTA*  |   25.31   |   0.851   |   0.184   |    5 minutes     |\n| OHTA** |   25.96   |   0.864   | **0.160** |    25 minutes    |\n\nwhere OHTA* means that it uses the same number of training steps as in our method, while that of OHTA** is times by five to ensure sufficient fitting. From these results, we can see that the computational complexity of our method is significantly lower than that of OHTA. The reduction of fine-tuning consumption comes from three aspects: (i) We choose to fine-tune the texture map bias instead of the whole network, which accelerates the process and lowers the calculation cost. (ii) OHTA designs a two-stage fine-tuning method while we only have one fine-tuning stage as texture map bias can be easily regularized. (iii) Thanks to the design of Gaussian Splatting, we lower the rendering points and reduce the rendering cost compared to NeRF-based methods.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
First, we thank ACs for organizing such a wonderful reviewing process and reviewers for their constructive comments that help to improve our paper greatly. We appreciate the confirmations from the reviewers on the proposed method, including Reviewer #NUjU "successfully applied the Gaussian splatting module for **the first time** in the problem of one-shot hand avatar creation", Reviewer #gWjw "The idea of the Interaction-aware Attention module is **novel**, which detect the interaction-aware geometric deformation in an intuitive way.", Reviewer #eWJx "The disentangled designs for one-shot reconstruction, using optimization-based identity maps for the one-shot stage, are **technically sound**.", and Reviewer #mjNr "**Sufficient technical novelty**". Reviewers also found this paper well-written and easy to follow (Reviewer #NUjU, #gWjw, and #mjNr). We hope this paper can shed light on GS-based one-shot 3D interacting hand reconstruction and be a considerable baseline.

We understand that the major concerns of reviewers are in-the-wild results and more comprehensive ablation studies. Hence, we provide two extra visual comparisons in the rebuttal file (denoted as RF), which include examples of in-the-wild results of three different tasks and four ablation studies. 

The two visual comparisons are summarized as follows: 

**Fig. 1** shows multiple examples of the proposed method which validate the effectiveness of our method in in-the-wild scenarios and justify the design for addressing out-of-distribution data. Three in-the-wild tasks are considered in this figure, including text-to-avatar (first four rows), in-the-wild reconstruction from real images (5th to 8th row), and texture editing (last row).

**Fig. 2** (top) demonstrates visual examples of shadow disentanglement (top) and qualitative results of four ablation studies (bottom). The visualization of shadow disentanglement includes a shaded image, an albedo image, and a shadow image for each pair of hands. This figure shows the shadow (dark) areas clearly, which suggests our method can disentangle albedo and shadow. 

**Fig. 2** (bottom) shows the results of four ablation studies, which can be further divided into the following four groups:

**1. Ablation study on single-hand images** (denoted as Ablation Hand Num.): We construct a single-hand baseline of the proposed method by masking one of the two hands in reference images. The results show that our method achieves better performance compared with the single-hand baseline, which is reasonable as two-hand images contain more complementary information.

**2. Ablation study on segmentation method** (Ablation Mask): To validate the impact of segmentation masks, we consider masks predicted by SAM and those rendered by ground-truth meshes. We observe that SAM masks enhance the performance of our model, since they are better aligned with hands and prevent background clutters.

**3. Ablation S1**: Ablation S1 shows the visual results of three different ablation settings: including (i) **the use of camera parameters**, which is a variant of our method without the camera parameters (denoted as w/o Cam.) and suffers from obvious performance drop; (ii) **adding shadow coefficient**, which incorporates shadow coefficient prediction into the proposed method (w/ Shadow); (iii) **Low Gaussian Points**, which reduces the number of Gaussian points to 24k to show that coarsen hand boundaries are caused by fewer numbers of Gaussian points.

**4. Ablation study on mesh estimation method** (Ablation Noise): We use hand parameters estimated by ACR to analyze how the accuracy of mesh reconstruction impacts the final results. The performance of our method with the predicted parameters is still satisfying, which suggests our method is robust to noise hand meshes to a certain extent. 

For the convenience of reading, **the quantitative results of the above studies are provided separately in our comments to each reviewer**. With these results, we hope all concerns have been addressed successfully.

The above revision will be incorporated into our paper and we are looking forward to comprehensive discussions in the next few days.

Meta-review: This is a borderline paper. Based on the initial ratings, and especially after the rebuttal, all reviewers are satisfied with the methodology, evaluation, and the responses provided by the authors.
Decision: Accept (poster)