HumanCrafter: Synergizing Generalizable Human Reconstruction and Semantic 3D Segmentation
Authors: Panwang Pan
Tingting Shen
Chenxin Li
Yunlong Lin
Kairun Wen
Jingjing Zhao
Yixuan Yuan
Venue: NeurIPS 2025 poster ()
Keywords: Gaussian Splatting
Semantic 3D Constructs
versatile view synthesis
Primary area: applications

Abstract: Recent advances in generative models have achieved high-fidelity in 3D human reconstruction, yet their utility for specific tasks (e.g., human 3D segmentation) remains constrained. We propose HumanCrafter, a unified framework that enables the joint modeling of appearance and human-part semantics from a single image in a feed-forward manner. Specifically, we integrate human geometric priors in the reconstruction stage and self-supervised semantic priors in the segmentation stage. To address labeled 3D human datasets scarcity, we further develop an interactive annotation procedure for generating high-quality data-label pairs. Our pixel-aligned aggregation enables cross-task synergy, while the multi-task objective simultaneously optimizes texture modeling fidelity and semantic consistency. Extensive experiments demonstrate that HumanCrafter surpasses existing state-of-the-art methods in both 3D human-part segmentation and 3D human reconstruction **from a single image**.

Human Synergizing Generalizable Human Reconstruction and Semantic 3D Segmentation
Panwang Pan Tingting Shen Chenxin Li Yunlong Lin?,
Kairun Wen Jingjing Zhao', Yixuan Yuan
Equal contribution Corresponning author
'ByteDance, Xiamen University, SCUHK
https://paulpannantt .eiggggggttt.h.. Lo/HumanCrafter
Abstract
Recent advances in generative models have achieved high-fidelity in 3D human reconstruction, yet their utility for specific tasks (e.g., human 3D segmentation) remains constrained. We propose HUMANCRAFTER, unified framework that enables the joint modeling of appearance and human-part semantics from a single image in feed-forward manner. Specifically, we integrate human geometric priors in the reconstruction stage and self-supervised semantic priors in the segmenta- tion stage. To address labeled 3D human datasets scarcity, we further develop an interactive annotation procedure for generating high-quality data-label pairs. Our pixel-aligned aggregation enables cross-tast ttoss-task synergy, while the multi-task objective simultaneously optimizes texture modeling fidelity and semantic con- sistency. Extensive experiments demonstrate that HUMANCRA, ATTER surpasses existing state-of-the-art methods in both 3D haman-part segmentation and 3D human reconstruction from a single image.
Introduction
Reconstructing high-fidelity 3D human representations and understanding human body and clothing attributes are a fundamental challenge in the 3D vision realm. The philosophy can unlock many novel and practical downstream applications, such semantic-guided 3D reasoning, context-aware human behavior analysis, and interactive semantic editing. This capability further facilitates immersive augmented and virtual reality (AR/VR), character stylization, and cinematic production.
In light of advances in Neural Radiance Fields [1], previous attempts [2, 3, have synthesized high-quality human novel views. Nevertheless, implicit representations are typically computationally intensive, as they rely on dense point querying in 3D space. Recent advances in 3D Gaussian Splatting (3DGS) [5] have provided real-time rendering for reconstructing high-quality explicit human models, which however rely on dense multi-view images [6. 7] or monocular video input [8, 10] and time- consuming per-subject optimization processes, limiting their stability and feasibility in downstream applications. With the emergence of large reconstruction models [11], recent advances can directly generalize the regression of 3D representations [12, 13, 14, 15] thanks to the prevalence of large-scale 3D object datasets [16]. However, in the specific task of human reconstruction, these works collapse and fail produce faithful and consistent novel views, primarily due to the scarcity of 3D human datasets and a lack of human prior knowledge as inductive bias in model design. Recent pioneering studies human reconstruction [17, 18] enable generalizable and robust synthesis under sparse- -iew settings. However, the crucial requirement for semantic 3D segmentation is currently hindered by the lack of expansive, well-labeled 3D human downstream task datasets. One workaround is first reconstruct and then leverage recent advances in 2D human visual foundation models [19]. This paradigm can result in extensive processing times and substantial engineering efforts. Furthermore,
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

20 Fazelaiks Medel (eg baplens)
Coder
Fraldeernd Fas Node
Couertry
200
Segueentttee
3D Cassina Radiance & Feature Field
Segmentation (Zoum [a)
Image Prompts to VersatileSplat
Test-Prumpla VeruaikkSplat
Editing Versatllesgiar
Iamerive ARVV Experience
mun ctiting agaient
wal"
Nea warriag
Tiet
Figure I: We introduce HUMANCRAFTER, a unified framework for simultaneous human 3D recon- struction and body-part segmentation from single images. HUMANCRAFTER introduces explicit 3D Gaussian VersatileSplats, showcasing enhanced performance over foundation models in delivering 3D-co 3D-consistent segmentation outcomes. This breakthrough offers significant advantages for down- stream applications.
2D operations cannot maintain 3D consistency and coherence across different viewpoints. The two- stage pipeline faces challenges including inconsistencies, high computational costs, and scalability issues, which hinder its robustness and efficiency. We hold the belief that the best way to understand something is to reconstruct it.
We introduce unified synthesis model that revolutionizes 3D reconstruction and editing through innovative view synthesis and semantic understanding. Our model takes a single image as input and generates explicit 3D Gaussian Splats enriched with semantic features, enabling human 3D reconstruction and body-part segmentation, while seamlessly enabling the 3D editing task and integrating into VR devices, as illustrated in Figure 1. Our approach builds upon the incorporation of tailored human priors and the aggregation of multi-view features with camera embeddings using Transformers. Specifically,
translate the set of aggregated features to pixel-aligned 3D Gaussians as initialized geometry. We unleash a pre-trained 2D model [20] into a 3D consistent feature field and establish weighting mechanism to propagate into multi-view, addressing the scarcity of labeled 3D semantic data. Finally, by jointly training on the constructed 3D segmentation datasets, which consist of 40,000 images from 2,500 human scans, HUMANCRAFTER enables novel view rendering and segmentation to mutually benefit from each other's 's task. In summary, our contributions can be summarized as follows:
We are first to introduce a unified 3D human representation and holistic framework that addresses versatile novel-view synthesis in feed-forward pass, allowing two independent tasks to mutually benefit.
HUMANCRAFTER leverages geometric human priors and the attention mechanism to ef- fectively bypass labor-intensive and computationally expensive steps, establishing new paradigm in the realm of human foundation models.
Experiments demonstrate that HUMANCRAFTER exhibits superior photorealistic 3D human Reconstruction and human-pan mannnntprttttoo segmentation capabilities, surpassing many state-of-the-art baselines simultancously with real-time rendering.
Experiments on in-the-wild images demonstrating HUMANCRAFTER strong generalizabil- ity to diverse image, and facilitating the potential real-world applications.

---

2 Related Work
2.1 Synergizing Reasoning with 3D Reconstruction.
2D human-centric models [21, 22, 19] exhibit remarkable performance through the advent of 2D foundation models and vision transformers [23, 20] and curated datasets (e.g.. Humans-300M [19]. Yet, these cutting-edge methods cannot achieve simultaneous 3D modeling and coherent segmentation due to lack 3D constraints. A crucial requirement for 3D reasoning is currently hindered by the paucity of extensive, well-annotated multi-view image datasets. Conversely, early studies like Semantic NeRF [24] and Panoptic Lifting [25] successfully embedded segmentation network semantic data into 3D scenes. 3DGS with Feature Field [26, 27, 28, 29] emerged prominent joint training approach for 3DGS and multiple prediction tasks. Recent works [30, 31] enable training-free 2D feature lifting to 3D paradigm for large scenes. However, the human ceman-specific domain remains largely unexplored, despite its potential applications in areas such as human editing, gaming, and film production.
2.2 Human Gaussian Splatting.
Recent advances [6. 32, 7] optimize photo-realistic animatable 3DGS from temporal multi-view images. In particular, HiFi4G [33] combines 3DGS with dual-graph mechanism to maintain spatial- temporal coherence, ASH [34] employs mesh UV parameterization for real-time rendering, and Animatable Gaussians [35] utilizes StyleUNet [36] and 3DGS for high-fidelity animatable avatars. For monocular video input [8, 37, 9, 38, 39, 40, 41, 42, 43, 44], human templates and Linear Blend Skinning are commonly adopted, necessitating per-instance optimization of 3DGS in a canonical space. For sparse-view scenarios, UPs-Gaussian [17] estimates depth maps from two-view stereo and unprojects them to pixel-wise 3D Gaussians. [45] further introduces regularization term and an epipolar attention mechanism to preserve geometry consistency between source views. EVA- Gaussian [46] employs a recurrent feature refiner address artifacts. [18] utilizes human templates for multi-scaffold photorealistic and accurate view rendering. For single image input, [47] incorporates generative diffusion models to predict triplane NeRF and followed by a ferward reconstruction model. Recent advances [48, 49, 50, 51] predicts the 3D outputs from a single input image in a generalizable manner through the combination of 2D Diffusion and well-conceived human priors. Notably, -luman-DDiffusil [51] introduces groundbreaking
soalle-mmage-t----- framework that synergizes diffusion models with 3D reconstruction to create highly consistent 3D avatars. Our work further enhances the synergy between 3D reconstruction and semantic segmentation. Our unified pipeline utilizes efficient nulti-view geometric aggregation and a refined Transformer module to significantly improve accuracy, real-time rendering, and multi-task consistency.
2.3 Generalizable Reconstruction Transformer.
GINA-3D, LRM and TripoSR [52, 53, 54] demonstrate that feed-forward Transformers trained on large-scale datasets [16, 55] are capable of generating 3D models in a generalizable manner by reconstructing triplane features for volumetric rendering. Real3D [56] further proposes self- training reconstruction framework that capitalizes on both synthetic and large-scale real-world datasets. Instant3D [57] proposes a two-stage pipeline, which first generates a sparse set of four posed images, and then directly regresses the Neural Radiance Fields (NeRF) [58]. [59] integrates efficient 3DGS with triplane representations, where point clouds inferred from input images query triplane features to decode 3DGS attributes. CRM, MeshLRM, and InstantMesh [60, 61, 62] replace the triplane NeRF with FlexiCubes [63] outputs, incorporating differentiable mesh extraction and rendering for potential applications. LGM, GRM, and GS-LRM [64, 65, 66) predict pixel-wise 3DGS parameters [67, 68] from multiple images, enabling scalable reconstruction frameworks. However, such methods lack human-specific priors as tailored inductive biases in Transformers, failing to synthesize faithful novel views, especially reconstructing human body details.
3 Method
Preliminary of 3D Gaussians Splatting. 3DGS [5] formulates the 3D representation collection of NN Gaussian primitives {G,) IN Each G, is characterized by an opacity o E R. a mean location
Mp E R²3, scaling factor s, E 3 an orientation quaternion q, E R2,, and color feature cp E RC
3

---

Encoder
Xdy
Tolien
FDM
Pauchity
Radiunee Poedd
Nets
Puichify
FEN
Difhaion Midil
Add Norm
Punur Agpapitic
Macker Raya Yonts
30 Conscian Prinktive()))
30 Feature Fichd
Figure 2: The network architecture of CRAFTERR The proposed method fully utilizes 2D diffusion priors and human body geometry features to regress pixel-aligned point maps via generic Transformer (Sec. 3.1). Subsequently, another Transformer (Sec. 3.2) employs attention mechanism to produce a set semantic 3D Gaussians that encapsulate geometric, appearance, and semantic information. The entire pipeline is trained in end-to-end manner by minimizing loss function (Sec. 3.3) that compares the predicted outputs against ground truth data and rasterized label maps from novel viewpoints.
maintained for rendering, where spherical harmonics (SH) model view fiewddependent effects.
Specifically, each Gaussian can be formulated as: G,(x) exp (- - (pp) Σ (x 1kp)
The rendering process involves projecting the 3D Gaussians onto the image plane as 2D Gaussians and performing alpha blending for each pixel in front-to-back depth order, thereby determining the final color, opacity, and depth maps.
Pipeline Overview. Given a single RGB image I E H W Wx3 our objective is to jointly synthesize multiple properties novel views using the multi-task labels from the given source views without any optimization or fine-tuning. To accommodate versatile labels, we extend the 3D Gaussians Gp to construct feature field, enabling the rendering of feature maps through differentiable rasterizer.
To this end, we firstly propose purely Transformer (Sec. 3.1) to aggregate multi-view input images, followed by attention mechanism (Sec. 3.2) to facilitate downstream tasks. Furthermore, we propose a multi-tase task loss (Sec. 3.3) to achieve high-fidelity texture rendering. An overview of the model architecture is illustrated in Figure 2.
3.1 Human Prior for Feature Aggregation
Image Diffusion Prior. We leverage the pre-trained 2D diffusion model SV3D [69] as appearance
prior. For the input image Io, we leverage CLIP image encoder [70] obtain the image embedding e as conditions. Subsequently, we progressively denoises gaussian noises into temporal-continunt
N
multi-view images by spatial-temporal UNet D, [71]. In this work, also employ the SMPL [72] as the 3D human parametric model to guide the Feature Aggregation. Inspired by [73, 74], we render the side-view normal images guide, which are then concatenated with the input image and corresponding Plücker embeddings [75] along the channel dimension, resulting in dense pose- conditioned images. These pose-conditioned images divided into non-overlapping patches [23] and mapped to di dimensional patch tokens F, x by Linear layer, where h H/P and w W represent the height and width of the feature map, respectively, and denotes the patch size.
Cross-view Attention Module. For feature interaction within patch tokens, we utilize N, layers of Grouped Query Attention blocks [76] with RMS neeeoommllzatinn, GELU activation, and feed- forward network. These features are subsequently mapped into the location of 3D Gaussian Splatting. To accurately model the positioning of Gaussians, we predict the depth map D; HR for each input image and additional 3D positional offset A, HHxWxs x3, A pixel located (u, v) in the image I, is unprojected from the image plane onto the 3D position M, 1 by unproject function

---

II II (K;R,;
(I) where, Di(u, u] represents the depth value at pixel (u, v) in the i-th view's depth map; R, is the rotation matrix of the i-th view, K is the camera intrinsic matrix, and t, the translation vector of the i-th view. The proposed Transformer architecture leverages feature aggregation to synergistically exploit human gcometry priors and the information embedded within input images, effectively bridging the 2D feature space and 3D coordinates to deliver enriched geometric representations.
[u, = R.R.K.D.((L... o| t; + A [u,
3.2 Self-Supervised Model as Inductive Bias
We employ the DINOv2 wits14-with-regie proven effective for 3D scene segmentation [78] and diverse downstream tasks [79]. We freeze the network parameters and extract features from the input image, denoted as f; E R (hxx))xd,, where d2 is the dimension of the image encoder. Furthermore, as the feature aggregation stage for the posed images has already learned the relevance between each position token, we directly extract the attention weights from the previous Transformer and perform weighted combination of the features which is independent of the feature dimension. We leverage the tailored inductive biases learned by the preceding Transformer across the hierarchical Transformer blocks. Specifically, for each Transformer block, the feature-wise similarity is extracted by the dot product of Q Q(F,) and (FF,) corresponding each ((f.).
[20, 77] 2D Image Encoder, which has been
(QF.)K(F.) vd
CrossAttm(f) == SoftMax
+B).
(2)
where, de presents the dimension of F,, B is the relative position bias. Ultimately, from each output token f; xldx decode the attributes of pixel-aligned Gaussians G in the corresponding patch using a convolutional layer with a x kernel. The final pixel color is calculated by blending N ordered Gaussians overlapping the pixels via the following rendering function:
j)). This equation efficiently models the contributions of each Gaussian the pixel's final appearance. accounting for their transparency and layering order. To facilitate body-part semantic 3D representation, inspired by [26]. we augment the 3D Gaussians Splats with a learnable semantic feature embedding (denoted as 3D Gaussians primitives) and rasterize onto the 2D image plane by blending Gaussians that overlap with each pixel using feature rendering function. This implies that the novel view synthesis task and human-part segmentation task share the same 3D Gaussian parameters, where N is the number of Gaussian primitives participating in the blending: E Mf, [1 o.)) where f indicates the final rasterized feature embedding the image plane, and f; represents the semantic
Ex C.O;; : [
3.3 Multi-task Training Objective
The 3D reconstruction loss function is designed to minimize the rendering loss Lrender for novel viewpoints. The loss for k, rendered multi-view images is defined as:
Crender :=Bi I,) + Cmack (M,, M.) + X CLPIPs((
(3)
where I; and i, denote the ground-truth images and rendered images via 3D Gaussian Splatting, respectively. M, and M, represent the original and rendered foreground masks. Lmse measures the mean squared error loss, and Cp measures the perceptual loss [80]. Am and Xp are hyperparameters employed for balancing the respective loss terms. Building upon prior work [31], the loss function for the feature field is minimized during training by utilizing rasterized feature maps novel views f and directly inferred feature maps using ground truth images on new views f, thereby facilitating the learning blending weights for consistent semantic field regression. Laist 1 CosSim f)
1 f-f where CosSim(- (---) denotes the cosine similarity between the predicted and ground truth feature maps, which serves as a distance metric be minimized during training. Finally, we construct semantic segmentation dataset for fine-tuning. We employ 28 classes for body-part segmentation,
||]||
5

---

along with the background class, following [19]. We jointly train the network on all tasks using differentiable rendering. Our model can be optimized in an end-to-end manner:
= iE iE(l,...k
E
(Rrender + Adia Ldis (f; fo) + Aseg" je(1,.....)
(S,
(4)
where @ is the model parameters, S; and SJ denote the annotated and predicted semantic segmenta- tion, respectively. TAEE represents the Cross Entropy Loss, and we enforce the predicted score to align with the ground truth viewpoint. Since only have small number of annotated viewpoints, LCE is added only when j E fl....,). The Aggregation mechanism is independent of features. Therefore, we can leverage 2D pretrained model to supervise the generation of rasterized novel views using Ldisa, and allow the two tasks to benefit each other through CCE.
4 Experiments
We conducted experiments on the following datasets evaluate the results of 3D human reconstruc- tion and segmentation. The 2D diffusion model takes approximately 6 seconds (with the number of views set to 2) to generate multi-view latent features, while the subsequent reconstruction stage takes only about 0.2 seconds.
Datasets. (I) THuman2.1 I Dataset [81] contains approximately 2500 human scans. Specifically, we select 2300 scans for training and the rest for evaluation. (2) 2K2K Dataset [82] includes 2000 human scans. Similarly, we select 1500 scans for training and the rest for evaluation. (3) Human MVImageNet [83] approximately comprises 4000 identities and 8000 outfits, which provide the rich musse-perspectives. Consistent with the protocol established in [48], we utilize PIXIE [84] as the SMPL parameter estimator, strategically placing 36 cameras across three hierarchical levels to capture full-body, upper-body, and facial views, with all renderings resolution of 512 512 pixels.
Curated Dataset. We randomly select 500 scans from the training dataset and annotate 8 semantic segmentation maps for each scan. More details are provided in Appendix A.I.
In-the-wild Dataset. To assess the model's generalizability under challenging conditions, we construct test dataset from Internet-soureed images. These images encompass diverse human poses, identities, and camera viewpoints.
Training Details. The hyperparameters Amask Ap are set to 1 and 0.1 in this paper. The hyperparame- ters Aua and Adisz are both set to 0.5. We use the AdamW optimizer with 31 = 0.9 and B2 0.95, and a weight decay of 0.05 is applied to all parameters except those in the LayerNorm layers. A cosine learning rate decay scheduler is employed, with a linear warm-up of 2,000 steps. The peak learning rate is set to 4 10-1, The training process is divided into two stages: the model is trained for 80K iterations at 256 x 256 resolution and then fine-tuned for an additional 20K iterations at 512 x 512 resolution. Please refer to Appendix A.2 for more detailed procedural insights.
4.1 Evaluation of Human 3D Segmentation
Comparisions. The semantic segmentation is evaluated by class-wise intersection over union (mloU) and average pixel accuracy (mAcc) on novel views as metrics. To provide more comprehensive evaluation of the algorithm's 3D consistency, we compare HUMANCRAFTER against the otate-of- the-art LSM [31] baseline. Additionally, we benchmark our approach against the 2D state-of-the-art
Table Comparison with feed-forward Human 3D Segmentation methods on 2K2K dataset.
2K2K
[82]
Method
Runtime
mIOU
Acc.
PSNR
SSIM
I.PPS
Two GT Inpul Views
LSM" [31]
0.724
0.873
23.811
0.892
0.053
108 ms object
Sapiens [19]
0.823
0.904
N/A
N/A
N/A
640 ms / frame
Ours
0.840
0.925
24.786
0.937
0.022
126 ms / object
Single View
HifmunDiillusionn [51] Sapiens [19]
0.781
0.851
21.832
0.891
0.069
23.21 / object
Ours
0.501
0.882
23.489
0.916
0.045
6.24 s/ object

---

Turget Viens
GT
Oar
Sapken
L.M
HuumactDDifsscinn Suplen
Saplan
Figure 3: Qualitative Results and Comparisons on Human 3D Segmentation on THuman2.1 and 2K2K datasets. HUMANCRAFTER achieves the best precise segmentation results in terms of 3D consistency.
human segmentation algorithm proposed in [19], which trained on large-scale human datasets. LSM* is trained using 3D human scans and 2D semantic segmentation maps, and it takes two images as input to ensure fair comparison. As illustrated in Table HUMANCRAFTER outperforms the state- of-the-art baselines in terms of segmentation accuracy, while exhibiting comparable reconstruction times. To enhance single-image reconstruction, we employ a two-stage approach using cutting- edge algorithms [51] baseline. As shown in Table 1, HUMANCRAFTER surpasses baselines in segmentation accuracy while maintaining comparable reconstruction times. Furthermore, Figure demonstrates the superior segmentation quality achieved by proposed method.
Evaluation of 3D Human Reconstruction
Comparisions. The quantitative assessment leverages metrics such PSNR. SSIM [87]. and VGG- LPIPS [80] to comprehensively analyze the rendering fidelity. Inspired by Diffsplat [], multi-view consistency is evaluated through COLMAP reconstructed point number [88]. We compare our approach with of-e-oofttheart methods. These include advanced reconstruction-batt basse techniques for bangle-image-ooditionnee generation: LGM [64], GRM [85], InstantMesh [62], Lara [86]. Hu- min3Diffusion [51], and PSHuman [49]. For the single-view setting, our method can either replicate the same input or utilize existing multi-view diffusion models available on the shelf to introduce 2D generative priors and achieve better results, demonstrated in the last row of Table 2. The single image-conditioned generation performance on the THuman 2. and 2K2K datasets evaluated in Table 2, with Qualitative results challenging scenarios (e.g., g., far camera viewpoints, complex human poses, and loose clothing) are presented in Figure and Figures in the Appendix. HU- MANCRAFTER demonstrates superior performance compared to -fate-of-the-ar baselines in both
Table 2: Comparison with feed-forward 3D reconstruction methods at a resolution of 512x512.
THuman2.1 [81]
2K2K
[82]
Method
PSNR
SSIMT
LPIPST
#Points
PSNRT
SSIM
LPIPS.
#Points
LGM [64]
20. 106
0.859
0.196
502.05
21.685
0.850
0.166
694.84
GRM [85]
20.503
0.868
0.141
602.46
21.496
0.858
0.171
722.95
InstantMesh [11]
19.997
0.875
0.128
1803.28
21.983
0.865
0.118
2028.30
LaRa [86]
18. 18.120
0.840
0.207
3035.43
19.113
0.860
0.207
4139.94
SIFU [73]
20.164
0.842
0.088
4500.68
21.698
0.904
0.084
4560. 18
PSHuman [49]
20.853
0.862
0.076
5321.23
21.932
0.892
0.076
4734.23
Human3Diffusinn [51]
22. 164
0.872
0.063
5123.52
22.323
0.882
0.053
5134.23
HumanCratter
23.186
0.907
0.046
5744.96
23.489
0.916
0.045
6453.23

---

Ii-the-wile Images
Input Image
Ours
Human3Difflsiio
PSHuman
SIFU
GRM
Figure 4: Novel-view images rendered by HUMANCRAFTER and the state-of oiiee-of-the-arr baselines on various datasets. Our method achieves the highest rendering quality. Please refer the zoomed-in regions for details.
rendering quality and 3D consistency metrics. Notably, LGM and GRM have been fine-tuned on our dataset to ensure fair comparison.
4.3
Ablation Studies
we carefully investigate the effectiveness of human prior and each design choice this subsection.
The Effect of Human Parametric Model. As shown in Table 3, the human geometric prior benefits significantly from the reconstruction model, particularly when k, of human body reconstruction tasks, rendering SMPL normals can provide richer geometric cues compared relying on coordinate or rendered depth maps. As the number of input viewpoints k, increases, the reconstruction model effectively resolves geometric ambiguities through stereo matching, reducing dependence on Human Pose Estimation. The statisties in Table validate HUMANCRAFTER consistently outperforms other variants in terms of image quality and fidelity.
Additionally, in the context
Model Design Choices. As demonstrated in Table 4 (b)-(d), the experiments confirm the efficacy of each crucial design decision. i) The HUMANCRAFTER model, in the absence of Plucker embeddings and relies solely on the SMPL-based geometric prior, demonstrates: a marginal decline in performance. ii) When integrating the pre-trained MAE model, denoted as ViT-s [89], in place of the DINOv2 model, HUMANCRAFTÉR exhibits slight performance decrease. This alteration attributed to
Table Ablation study of human geometry prior on 2K2K dataset.
Table 4: Ablation of model and objective design on 2K2K dataset.
PSNR
SSIM 1
LPIPS
#Param.
PSNR
SSIM
LPIPS +
(a) Full Model (Ours)
23.489
0.916
0.045
Depth
23.103
0.901
0.0270
42.4M
Ein.
23.327
(6) w/o DINOx2
22.025
0.891
0.055
Nermal (Ouns)
1
23-489
0916
0.045
42.4M
w/o Pixel-align Aggregation
21.183
41.2M
w/o Cas
22,464
0.896
0.055
with SMPL (Ours)
2
24.786
0.377
0.022
42.6M
(1 wfo Cen.
23.223
0.901
0.051

---

Imput Views
wio PA
with PA?
weo PA!
with PA
wio PA?
with PA!
wil PA7
with PA?
Figure 5: Ablation of Pixel-Align Aggregation. HUMAN CRAFTER with PA can leverage knowl- edge learned from novel-view synthesis task and incorporate pre-trained 2D model, thereby boosting semantic tasks.
the superior ability of the selected pre-trained model establish correspondences among the input view images, a crucial factor for pixel-aligne aggregation. iii) Furthermore, HUMANCRAFTER is solely enhanced with the Feature Aggregation Module akin to the current LGM methodology. and the learned features are directly forwarded through the GS-Decoder notable decline in performance observed, as illustrated Table (d) and Figure in the Appendix. This result underscores efficacy of the Pixel-align Aggregation Module.
The Effect of Loss Functions. As demonstrated Table 4 (c)-(D, without the LPIPS loss, novel view renderings are susceptible to blurriness and unnatural generations, leading to a slight performance decline. The integration of the distillation loss enhances 3D view consistency. Similarly, akin to LSM, the addition of the semantic distillation loss illustrates that integrating the human segmentation task enhances the performance of novel view synthesis.
4.4 Applications: Human Editing and Immersive Exploring
Figure and Figure 6 illustrate the potential scenarios enabled by the proposed model: (1) Text to VersatileSplats: ControlNet [90] is used produce human image with the human mask and text prompts. Subsequently, HUMANCRAFTED is employed to reconstruct 3D human from the generated image. (2) 3D Consistency Editing: HUMANCRAFTER'S 3D coherence and precise semantic masks are employed to direct the 3D human editing process, supported by a FLUX-based inpainting high efficiency, enabling real-time end-to-end 3D modeling. Following the generation of edited 3DGS primitives for the provided input views, seamless integration into VR devices.
model [91]. Immersive Exploring:
Figure 6: We demonstrate the generalizability of HUMANCRAFTER with in-the-wild images in challenging scenarios.

---

Figure 7: We demonstrate the generalizability of HUMANCRAFTER with in-the-wild images in challenging scenarios.
As depicted in Figure 4.3, we validate that Pixel-Align Aggregation effectively utilizes information from the task of synthesizing new viewpoints. As illustrated in Figure 6, potential application of our proposed model is its combination with the existing Text-to- Image (T2I) inpainting Diffusion model, such FLUX.1 [91]. Notably, the 3D Gaussian primitives VR devices. We demonstrate this through the application of watching high-fidelity virtual concert using the PICO 4 Ultra VR headset. As depicted in Figure 7, showcase the generalizability of our model using nn-the-wild images in challenging scenarios.
generate seamlessly integrate into
Conclusion
We have introduced HUMANCRAFTER, unified framework for 3D human reconstruction and understanding. First, we adopt tailored human priors and aggregate multi-view images from a 2D diffusion model and camera embedding features in Transformer. Second, translate the set of aggregated features to pixel-aligned 3D Gaussians as initialized geometry. We extend a 2D pre- trained model into 3D consistent feature field and establish weighting mechanism propagate into multi-view. Extensive experiments demonstrate that HUMANCRAFTER surpasses existing methods in terms of novel view synthesis quality and downstream task performance while exhibiting robustness in complex scenarios.
Broader Impacts HUMANCRAFTER allows users to generate 3D human models tailored their specific inputs, enabling a broad spectrum of downstream applications, such as AR/VR Chat, 3D cinematography, and 3D editing. However, this capability also presents potential ethical challenges, including privacy violations and racial biases. To mitigate these risks, it is imperative to establish robust ethical guidelines and enforce legal regulations.
Limitations and Future Works. Building on the significant acceleration our method provides for semantic 3D human reconstruction, compelling avenue for future work is its extension dynamic 4D scene generation with Gaussian representations [92]. Furthermore, by leveraging web-scale human datasets and relying solely on 2D supervision, extensive real-world video datasets could further unlock the potential of HÜMANCRAFTER. Moreover, there is potential for misuse, such as the arbitrary distribution of digital assets. These risks can be mitigated by embedding watermarks into the 3D assets [93, 94].
10

---

References
[1] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision (ECCV), 2020.
[2] Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields. Advances in Neural Information Processing Systems, 2020.
[3] Sida Peng, Yuanqing Zhang, Yinghao Xu, Qianqian Wang, Qing Shuai, Hujun Bao, and Xiaowei Zhou. Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9054 9063, 2021.
[4] Shoukang Hu, Fangzhou Hong, Liang Pan, Haiyi Mei, Lei Yang, and Ziwei Liu. Sherf: Generalizable human nerf from a single image. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) 2023.
[5] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkuchler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics (TOG), 2023.
[6] Wojeiech Zielonka, Timur Bagautdinov, Shunsuke Saito, Michael Zollhöfer, Justus Thies, and Javier Romero. Drivable 3d gaussian avatars. arXiv preprint arXiv:231 1.08881, 2023.
[7] Zhiyin Qian, Shaofei Wang, Marko Mihajlovic, Andreas Geiger, and Siyu Tang. 3dgs-avatar: Animatable avatars via deformable 3d gaussian splatting, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[8] Mengtian Li, Shengxiang Yao, Zhifeng Xie, Keyu Chen, and Yu-Gang Jiang. Gaussianbody: Clothed human reconstruction via 3d gaussian splatting. arXiv preprint arXiv:2401.0977 2024.
[9] Liangxiao Hu, Hongwen Zhang, Yuxiang Zhang, Boyao Zhou, Boning Liu, Shengping Zhang, and Liqiang Nie. Gaussianavatar: Towards realistic human avatar modeling from single video via animatable 3d gaussians. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[10] Sen Peng, Weixing Xie, Zilong Wang, Xiaohu Guo, Zhonggui Chen, Baorong Yang, and Xiao Dong. Rmavatar: Photorealistic human avatar reconstruction from monocular video based on rectified mesh-embedded gaussians. arXiv preprint arXiv:2501.071000 2025.
[11] Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. Lrm: Large reconstruction model for single image to 3d. arXiv preprint arXiv:2311.0m 2311.04000, 2023.
[12] Huiqun Wang, Yiping Bao, Panwang Pan, Zeming Li, Xiao Liu, Ruijie Yang, and Di Huang. relation distillation for unified 3d representation learning. In European Conference Computer Vision, pages 364-381. Springer, 2024.
[13] Chenguo Lin, Panwang Pan, Bangbang Yang, Zeming Li, and Yadong Mu. Diffsplat: Re- purposing image diffusion models for scalable gaussian splat generation. arXiv preprint urXiv:2501.16766 2025.
[14] Yuchen Lin, Chenguo Lin, Panwang Pan, Honglei Yan, Yiqiang Feng, Yadong Mu, and Katerina Fragkiadaki. Parterafter: Structured 3d mesh generation via compositional latent diffusion transformers. arXiv preprint arXiv:2506.055 0055500005555,.. 2025.
[15] Chenguo Lin, Yuchen Lin, Panwang Pan, Xuanyang Zhang, and Yadong Mu. Instructlayout: Instruction-driven 2d and 3d layout synthesis with semantic graph prior. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025.
[16] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.

---

[17] Shunyuan Zheng, Boyao Zhou, Ruizhi Shao, Boning Liu, Shengping Zhang, Liqiang Nie, and Yebin Liu. Gps-gaussian: Generalizable pixel-wise 3d gaussian splatting for real-time human novel view synthesis. In Proceedings of the IEEF/CVF Conference Computer Vision and Puttern Recognition (CVPR), 2024.
[18] Youngjoong Kwon, Baole Fang. Yixing Lu, Haoye Dong, Cheng Zhang, Francisco Vicente Carrasco, Albert Mosella-Montoro, Jianjin Xu, Shingo Takagi, Daeil Kim, et al. General- izable human gaussians for sparse view synthesis. In European Conference on Computer Vision(ECCV), pages
Springer, 2025.
[19] Rawal Khirodkar, Timur Bagautdinov, Julieta Martinez, Su Zhaoen, Austin James, Peter Selednik, Stuart Anderson, and Shunsuke Saito. Sapiens: Foundation for human vision models. In European Conference on Computer Vision, pages 206-228. Springer, 2025.
[20] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo. Mare Szafraniec, Vasil Khalidov, Pierre Fernandez. Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint
2023.
[21] Yuanzheng Ci, Yizhou Wang, Meilin Chen, Shixiang Tang, Lei Bai, Feng Zhu, Rui Zhao, Fengwei Yu, Donglian Qi, and Wanli Ouyang. Unihep: A unified model for human-centric perceptions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17840-17852, 2023.
[22] Yizhou Wang, Yixuan Wu, Shixiang Tang, Weizhen He, Xun Guo, Feng Zhu, Lei Bai, Rui Zhao, Jian Wu, Tong He, et al. Hulk: A universal knowledge translator for human-centric tasks. arXiv preprint arXiv:2312.01167, 8, 2023.
[23] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR), 2020.
[24] Shuaifeng Zhi, Tristan Laidlow, Stefan Leutenegger, and Andrew Davison. In-place scene la- belling and understanding with implicit scene representation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15838-15847, 2021.
[25] Yawar Siddiqui, Lorenzo Porzi, Samuel Rota Buló, Norman Müller, Matthias Nieβner, Angela Dai, and Peter Kontschieder. Panoptic lifting for 3d scene understanding with neural fields. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 9043-9052, 2023.
[26] Shijie Zhou, Haoran Chang, Sicheng Jiang, Zhiwen Fan, Zehao Zhu, Dejia Xu, Pradyumna Chari, Suya You, Zhangyang Wang, and Achuta Kadambi. Feature 3dgs: Supercharging 3d gaussian splatting to enable distilled feature fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 21676-21685, 2024.
[27] Jun Guo, Xiaojian Ma, Yue Fan, Huaping Liu, and Qing Li. Semantic gaussians: Open- vocabulary scene understanding with 3d gaussian splatting, arXiv preprint 4rX1v:2403.15624, 2024.
[28] Mingqiao Ye, Martin Danelljan, Fisher Yu, and Lei Ke. Gaussian grouping: Segment and edit anything in 3d scenes. In European Conference Computer Vision, pages 162-179, Springer, 2025.
[29] Renjie Li, Zhiwen Fan, Bohua Wang, Peihao Wang, Zhangyang Wang, and Xi Wu. Versatile- gaussian: Real-time neural rendering for versatile tasks using gaussian splatting. 2025.
[30] Juliette Marrie, Romain Ménégaux, Michael Arbel, Diane Larlus, and Julien Mairal. Ludvig: Learning-free uplifting of 2d visual features to gaussian splatting scenes. arXiv preprint 0rXiv:2410.14400 2024.
12

---

[31] Zhiwen Fan, Jian Zhang, Wenyan Cong, Peihao Wang, Renjie Li, Kairun Wen, Shijie Zhou, Achuta Kadambi, Zhangyang Wang, Danfei Xu, al. Large spatial model: End-to-end unposed images to semantic 3d. In Advanced in Neural Information Processing Systems (NeurIPS), 2024.
[32] Arthur Moreau, Jifei Song, Helisa Dhamo, Richard Shaw, Yiren Zhou, and Eduardo Pérez- Pellitero. Human gaussian splatring: Real-time rendering of animatable avatars. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[33] Yuheng Jiang, Zhehao Shen, Penghao Wang, Zhuo Su, Yu Hong, Yingliang Zhang, Jingyi Yu, and Lan Xu. Hifi4g: High-fidelity human performance rendering via compact gaussian splatting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[34] Haokai Pang, Heming Zhu, Adam Kortylewski, Christian Theobalt, and Mare Habermann. Ash: Animatable gaussian splats for efficient and photoreal human rendering, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[35] Zhe Li, Zerong Zheng, Lizhen Wang, and Yebin Liu. Animatable gaussians: Learning pose- dependent gaussian maps for high-fidelity human avatar modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[36] Lizhen Wang, Xiaochen Zhao, Jingxiang Sun, Yuxiang Zhang, Hongwen Zhang, Tao Yu, and Yebin Liu. Styleavatar: Real-time photo-realistic portrait avatar from a single video. In ACM SIGGRAPII 2023 Conference Proceedings, pages 1-10, 2023.
[37] Shoukang Hu and Ziwei Liu. Gauhuman: Articulated gaussian splatting from monocular human videos. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[38] Xiaozheng Zheng, Chao Wen, Zhaohu Li, Weiyi Zhang, Zhuo Su, Xu Chang, Yang Zhao, Zheng Lv, Xiaoyuan Zhang, Yongjie Zhang, Guidong Wang, and Lan Xu. Headgap: Few-shot 3d head avatar via generalizable gaussian priors, 2024.
[39] Mingwei Li, Jiachen Tao, Zongxin Yang, and Yi Yang. Human101: Training 100+ fps human gaussians in 100s from 1 view. arXiv preprint ar arXiv:2312.15255 2023.
[40] Xinqi Liu, Chenming Wu, Jialun Liu, Xing Liu, Chen Zhao, Haocheng Feng, Errui Ding, and Jingdong Wang. Gva: Reconstructing vivid 3d gaussian avatars from monocular videos. arXiv preprint arXiv:2402. 16607, 2024.
[41] Zhijing Shao, Zhaolong Wang, Zhuang Li, Duotun Wang, Xiangru Lin, Yu Zhang, Mingming Fan, and Zeyu Wang. SplattingAvatar: Realistic Real-Tima Human Avatars with Mesh- Embedded Gaussian Splatting. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2024.
[42] Jing Wen, Xiaoming Zhao, Zhongzheng Ren, Alexander G Schwing, and Shenlong Wang. Gomavatar: Efficient animatable human modeling from monocular video using gaussians- on-mesh. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2024.
[43] Jiahui Lei, Yufu Wang, Georgios Pavlakos, Lingjie Liu, and Kostas Daniilidis. Gart: Gaussian articulated template models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[44] David Svitov, Pietro Morerio, Lourdes Agapito, and Alessio Del Bue. Haha: Highly articulated gaussian human avatars with textured mesh prior. arXiv preprint arXiv:2404.01055 2024.
[45] Boyao Zhou, Shunyuan Zheng, Hanzhang Tu, Ruizhi Shao, Boning Liu, Shengping Zhang, Liqiang Nie, and Yebin Liu. Gps-gaussian+: Generalizable pixel-wise 3d gaussian splatting for real-time human-scene rendering from sparse views. arXiv preprint arXiv:2411 I 1363, 2024.
13

---

[46] Yingdong Hu, Zhening Liu, Jiawei Shao, Zehong Lin, and Jun Zhang. Eva Sva-gaussian: 3d gaussian- based real-time human novel view synthesis under diverse camera settings. arXiv preprini arXiv:2410001425, 2024.
[47] Zhenzhen Weng, Jingyuan Liu, Hao Tan, Zhan Xu, Yang Zhou, Serena Yeung-Levy. and Jimei Yang. Template-free single-view 3d human digitalization with crffusiio-guuidee Irm. arXiv preprint arXiv:2401. 12/7,,1151 2024.
[48] Panwang Pan, Zhuo Su, Chenguo Lin, Zhen Fan, Yongjie Zhang, Zeming Li, Tingting Shen, Yadong Mu, and Yebin Liu. Humansplat: Generalizable single-image human gaussian splatting with structure priors. In Advanced Neural Information Processing Systems (NeurIPS), 2024.
[49] Peng Li, Wangguandong Zheng, Yuan Liu, Tao Yu, Yangguang Li, Xingqun Qi, Mengfei Li, Xiaowei Chi, Siyu Xia, Wei Xue, et al. Pshuman: Photorealistic single-view human reconstruction using cross-scale diffusion. arXiv preprint arXiv:2409. 24111011,,, 2024.
[50] Yiyu Zhuang, Jiaxi Lv, Hao Wen, Qing Shuai, Ailing Zeng, Hao Zhu, Shifeng Chen, Yujuu Yang, Xun Cao, and Wei Liu. Idol: Instant photorealistic 3d human creation from a single image, 2025.
[51] Yuxuan Xue, Xianghui Xie, Riccardo Marin, and Gerard Pons-Moll. Human 3diffusion: Realis- tic avatar creation via explicit 3d consistent diffusion models. arXiv preprint 28X4::2406.08475, 2024.
[52] Bokui Shen, Xinchen Yan, Charles R Qi, Mahyar Najibi, Boyang Deng, Leonidas Guibas, Yin Zhou, and Dragomir Anguelov. Gina-3d: Learning to generate implicit neural assets in the wild. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2023.
[53] Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. Lrm: Large reconstruction model for single image to 3d. In International Conference on Learning Representations (ICLR), 2024.
[54] Dmitry Tochilkin, David Pankratz, Zexiang Liu, Zixuan Huang, Adam Letts, Yangguang Li, Ding Liang, Christian Laforte, Varun Jampani, and Yan-Pei Cao. Triposr: Fast 3d object reconstruction from a single image. arXiv preprint arXiv:2403.000 00111.251.1... 2024.
[55] Tong Wu, Jiarui Zhang, Xiao Fu, Yuxin Wang, Jiawei Ren, Liang Pan, Wayne Wu, Lei Yang, Jiaqi Wang. Chen Qian, al. Omniobject3d: perception, reconstruction and generation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
3d object dataset for realistic
[56] Hanwen Jiang, Qixing Huang, and Georgios Pavlakos. Real3d: Scaling up large reconstruction models with real-world images. arXiv preprint arXiv:2406.0844 2024.
[57] Jiahao Li, Hao Tan, Kai Zhang, Zexiang Xu, Fujun Luan, Yinghao Xu, Yicong Hong, Kalyan Sunkavalli, Greg Shakhnarovich, and Sai Bi. Instant3d: Fast text-to-3d with sparse-view gener- ation and large reconstruction model. In International Conference on Learning Representations (ICLR), 2024.
[58] Zirui Wang, Shangzhe Wu, Weidi Xie, Min Chen, and Victor Adrian Prisacariu. Nerf-: Neural radiance fields without known camera parameters. arXiv preprint Xiiv:2102.07066 2021.
[59] Zi-Xin Zou, Zhipeng Yu, Yuan-Chen Guo, Yangguang Li, Ding Liang, Yan-Pei Cao, and Song-Hai Zhang. Triplane meets gaussian splatting: Fast and generalizable single-view 3d reconstruction with transformers. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[60] Zhengyi Wang, Yikai Wang, Yifei Chen, Chendong Xiang, Shuo Chen, Dajiang Yu, Chongxuan Li, Hang Su, and Jun Zhu. Crm: Single image to 3d textured mesh with convolutional reconstruction model. European Conference Computer Vision 2024.
14

---

[61] Xinyue Wei, Kai Zhang, Sai Bi, Hao Tan, Fujun Luan, Valentin Deschaintre, Kalyan Sunkavalli, Hao Su, and Zexiang Xu. Meshlrm: Large reconstruction model for high-quality mesh. arXiv preprint arXiv:2404. 12385, 2024.
[62] Jiale Xu, Weihao Cheng, Yiming Gao, Xintao Wang, Shenghua Gao, and Ying Shan. In- stantmesh: Efficient 3d mesh generation from a single image with sparse-view large recon- struction models. arXiv preprini urXiv:2404.07191 2024.
[63] Tianchang Shen, Jacob Munkberg, Jon Hasselgren, Kangxue Yin, Zian Wang, Wenzheng Chen, Zan Gojcic, Sanja Fidler, Nicholas Sharp, and Jun Gao. Flexible isosurface extraction for gradient-based mesh optimization. ACM Transactions on Graphics (TOG), 2023.
[64] Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, and Ziwei Liu. Lgm: Large multi-view gaussian model for high-resolution 3d content creation. European Conference on Computer Vision (ECCV), 2024.
[65] Yinghao Xu. Zifan Shi, Wang Yifan, Hansheng Chen, Ceyuan Yang. Sida Peng, Yujun Shen, and Gordon Wetzstein. Grm: Large gaussian reconstruction model for efficient 3d reconstruction and generation. arXiv preprint arXiv:2403. 14621, 2024.
[66] Kai Zhang. Sai Bi, Hao Tan, Yuanbo Xiangli, Nanxuan Zhao, Kalyan Sunkavalli, and Zexiang Xu. Gs-Irm: Large reconstruction model for 3d gaussian splatting. In European Conference Computer Vision (ECCV), 2024.
[67] Stanislaw Szymanowicz, Christian Rupprecht, and Andrea Vedaldi. Splatter image: Ultra-fast single-view 3d reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[68] David Charatan, Sizhe Li, Andrea Tagliasacchi, and Vincent Sitzmann. pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2024.
[69] Vikram Voleti, Chun-Han Yao, Mark Boss, Adam Letts, David Pankratz, Dmitry Tochilkin, Christian Laforte, Robin Rombach, and Varun Jampani. Sv3d: Novel multi-view synthesis and 3d generation from single image using latent video diffusion. European Conference on Computer Vision (ECCV), 2024.
[70] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on Machine Learning (ICML), 2021.
[71] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Do- minik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127 2023.
[72] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. Smpl: A skinned multi- culli-person linear model. ACM transactions on graphics (TOG), 2015.
[73] Zechuan Zhang, Zongxin Yang, and Yang. Sifu: Side-view conditioned implicit function for real-world usable clothed human reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2024.
[74] Ruizhi Shao, Youxin Pang. Zerong Zheng, Jingxiang Sun, and Yebin Liu. Human4dit: Free- view human video generation with 4d diffusion transformer. arXiv preprint arXiv:2405.17400 2024.
[75] Julius Plücker. Xvii. on new geometry of space. Philosophical Transactions of the Royal Society of London, 1865.
[76] Joshua Ainslie, James Therph Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. arXiv preprint 3:31v:2305.13245, 2023.
15

---

[77] Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. International Conference on Learning Representations (ICLR). 2024.
[78] Zhiwen Fan, Peihao Wang, Yifan Jiang, Xinyu Gong, Dejia Xu, and Zhangyang Wang. Nerf-sos: Any-view self-supervised object segmentation on complex scenes, arXiv preprint arXiv:2209.0877 2022.
[79] Junyi Zhang, Charles Herrmann, Junhwa Hur, Luisa Polania Cabrera, Varun Jampani, Deqing Sun, and Ming- Hsuan Yang. A tale of two features: Stable diffusion complements dino for zero-shot semantic correspondence. Advances in Neural Information Processing Systems, 36, 2024.
Ming-Hsuan
[80] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreason- able effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[81] Tao Yu, Zerong Zheng, Kaiwen Guo, Pengpeng Liu, Qionghai Dai, and Yebin Liu. Function4d: Real-time human volumetric capture from very sparse consumer rgbd sensors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[82] Sang-Hur Hun Han, Min-Gyu Park. Ju Hong Yoon, Ju-Mi Kang, Young-Jae Park, and Hae-Gor Jeon. High-fidelity 3d human digitization from single 2k resolution images. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pottern Recognition (CVPR), 2023.
[83] Zhangyang Xiong, Chenghong Li, Kenkun Liu, Hongjie Liao, Jianqiao Hu, Junyi Zhu, Shu- liang Ning, Lingteng Qiu, Chongjie Wang, Shijie Wang. et al. Mvhumannet: A large-scale dataset of multi-view daily dressing human captures. In Proceedings of the IEEE/CVF Confer- ence Computer Vision and Pattern Recognition, pages 19801-19811, 2024.
[84] Yao Feng, Vasileios Choutas, Timo Bolkart, Dimitrios Tzionas, and Michael J Black. Collabo- rative regression of expressive bodies using moderation. In International Conference on 3D Vision (3DV), 2021.
[85] Yinghao Xu, Zifan Shi, Wang Yifan, Hansheng Chen, Ceyuan Yang, Sida Peng, Yujun Shen, and Gordon Wetzstein. Grm: Large gaussian reconstruction model for efficient 3d reconstruction and generation. arXiv preprint arXiv:2403. 14621, 2024.
[86] Anpei Chen, Haofei Xu, Stefano Esposito, Siyu Tang, and Andreas Geiger. Lara: Efficient large-baseling radiance fields, 2024.
[87] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing (TIP), 2004.
[88] Johannes I. Schhonberger and Jan-Michael Frahm. ings of the IEEE conference on computer vision and pattern recognition, pages 4113-4113, 2016.
revisited. In Proceed-
[89] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners, 2021.
[90] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 3836-3847, 2023.
[91] Black Forest Labs.
2023.
[92] Renjie Li, Panwang Pan, Bangbang Yang. Dejia Xu, Shijie Zhou, Xuanyang Zhang, Zeming Li, Achuta Kadambi, Zhangyang Wang, Zhengzhong Tu, al. 4k4dgen: Panoramic 4d generation at 4k resolution. arXiv preprint arXiv:2406. 13527, 2024.
[93] Chenxin Li, Brandon Y Feng, Zhiwen Fan, Panwang Pan, and Zhangyang Wang. Stegan- erf: Embedding invisible information within neural radiance fields. In Proceedings of the IEEE/CVF international conference computer vision, pages 441-453, 2023.
16

---

[94] Chenxin Li, Hengyu Liu, Zhiwen Fan, Wuyang Li, Yifan Liu, Panwang Pan, and Yixuan Yuan. Instantsplamp: Fast and generalizable stenography framework for generative gaussian splatting. In The Thirteenth International Conference on Learning Representations, 2025.
[95] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan Yen Lo, et al. Segment anything. In Proceedings the IEEE/CVF International Conference Computer Vision, pages 2023.
[96] Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691, 2023.
[97] Benjamin Lefaudeux, Francisco Massa, Diana Liskovich, Wenhan Xiong, Vittorio Caggiano, Sean Naren, Min Xu, Jieru Hu, Marta Tintore, Susan Zhang. Patrick Labatut, Daniel Haziza, Luca Wehrstedt, Jeremy Reizenstein, and Grigory Sizov. xformers: modular and hack- able transformer modelling library. 2022.
[98] Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. arXiv preprint arXiv: Ttt4::660006177 2016.
[99] Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. Mixed precision training. In International Conference on Learning Representations (ICLR), 2018.
[100] Baowen Zhang, Chuan Fang, Rakesh Shrestha, Yixun Liang, Xiaoxiao Long, and Ping Tan. Rade-gs: Rasterizing depth in gaussian splatting, 2024.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of the work discussed in Section ??.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
18

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The paper fully discloses the information needed to reproduce the main experimental results of the paper.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
19

---

Answer: [No]
Justification:
Code will be released upon acceptance of the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
www.aaimissioooooooooooooo oolicyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ttaai//ggiddeeeeeeuubtiiiii / //npp..c.//// wttpccccppppccccceeec.ccccccccct haabbiicgguidss..e/ee /Codeeeooiiisaeeeeeee cesteummisiioooi wwweinsssiooooooo ceeinniyycy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
All the training and test details are specified in Section ?? and Section ??.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Please refer to Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
20

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer to Section B.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http///eeer/..ee/eee/eee cc Bew.ei///thiiiiiiieeeeee dereeGuddelinng lines?
Answer: [Yes]
Justification: The research conducted in the paper conforms, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please refer Section 5.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
21

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Creators or original owners assets used in the paper are properly credited, and the license and terms of use are explicitly mentioned and properly respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
22

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
23

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https : http///eenn../eeeeee/eeee httpe//rps..eeeooeeeeeeeeeeeeeee http.////oooeeeeeee////////1//1 for what should or should be described.
24

---

A More Implementation Settings
We provide comprehensive implementation details this section to facilitate the reproducibility of our work. Specifically, in Section. A. , we provide the details of how to construct semantic segmentation data. Section. A.2, we provide details about training details. In Section. B, we offer further explanation of the implementation for HUMANCRAFTER.
A.1 Constructed Dataset Details
Interactive Annotation. Human-part Semantic Segmentation aims to classify pixels in the input image I, into Neins categories while ensuring 3D consistency. Following Sapiens [19], we construct dataset with Nelass = 28 (27 body parts and one background class). The class names are as follows: 'Background'. 'Apparel', Face _eck', 'Hair', 'Lot',Foot', 'Left Hand", Lower Arm', "Leet_Lowwr__eg" 'Left_Shoe, 'Left_Sock', Upper_Upper "Left_Upper_Leg" 'Lower_Clotee "eottr_Clohhig'' 'Right_Foot', 'Right_ "Right_Hand; Lowerr Sught_UpperAAm' Right_Upper 'Torso', "Upper.Clothing 'Lower 'Upper.Lip', 'Lethr.Teeth', 'Upper_Teett and 'Tongue'.
Loger__Lowrr_Leg 'Right,Shoe', Right_Sook',,
To accelerate the manual annotation process, we utilize the Segment Anything Model (SAM) [95] for assisted labeling. The dataset construction process is illustrated in Figure In (a), we show the instance data and segmentation pipeline, and in (b), we demonstrate how to accelerate annotation using Segment Anything Model.
k << <<>>>>
Poot
Reforeeen
Mask Decoder
Heracive
Prompt Eacader
Encoder
Nogetvve
Posttie Prioc
(a)
Figure 8: The dataset construction pipeline. (a) The instance and semantic segmentation annotation pipeline allows us to repeatedly reuse the features of input images and the prior negative and positive coordinates. (b) Accelerating annotation procedure by leveraging the Segment Anything Model [95].
A.2 More Training Details
To accelerate the training process, we employ Flash-Attention-v22 [96] from the xFormers library [97], gradient checkpointing [98), and BFloat16 mixed-precision arithmetic [99]. Leveraging a pre-trained model and human geometric priors, our method takes 7 days of training on 8 NVIDIA A800 GPUs.
Differentiable 3DGS Rasterization. A modified 3DGS rasterization implementation' [100] sup- ports depth, alpha, and normal rendering. Additionally, we extend 3DGS rasterization pipeline to incorporate feature attributes, enabling feature rendering from perspectives, where all operations are differentiable. Due to limitations in GPU memory, we render 3DGS features with a dimension- ality of up to 1024 at most. Initially, filter out low-opacity 3D Gaussian splats (a, 0.005) to enhance rendering speed without compromising quality.
'nttps://githin Bttpgthtw...eeooooe//Bee-e--- git
25

---

B More Details of HUMANCRAFTER
Dataset Normalization. To better learn the 3DGS attributes. we place the human seans at the origin of the coordinate system and normalize them to unit cube, so that they are located within the bounding box ([-1, 1]3) the world space. The camera poses of the rendered views are normalized with global scale of 1.4.
3D Gaussian Primitives Normalization. As 3D Gaussians are unstructured explicit representation the parameterization of the output parameters can affect the model's 's convergence. For numerical values of Gaussian splat properties, we set to Spherical Harmonics 3, and all attributes confined within [0, 1] in preparation of diffusion-based generation, outputs of 3DGS are all activated by the sigmo d function, except for r, which Ln-normalized to yield unit quaternions. RGB color and opacity are already supposed to be in [0, 1]. Raw scale $ is linearly interpolated with predefined values 8min and 8max [65]. s = Smin sigmoid + 8max (1 s sigmoid($)) Here, 8min and 8max set to 5e-4 and 2e-2 respectively to represent fine- tnn-grain details.
Ablation on Image Encoder [20]. We freeze the image encoders based on DINOv2's s best practices to leverage its pre-trained features and to maintain training efficiency by only training lightweight decoder. We validated this design choice with an ablation study on the 2K2K dataset, as shown in Table 5. The results indicate that fine-tuning the image encoder provides only marginal gains (+0.032 PSNR).
Table 5: Ablation study the effect of fine-tuning the DINOv2 image encoder. The experiment is conducted on the 2K2K dataset.
Method
PSNR
SSIM
LPIPS +
Fine-tumed DINOv2
23.521
0.918
0.046
Frozen DINOv2
23.489
0.916
0.045
Ablation Tual-Transformn The first Transformer (Feature Aggregation) focuses the general task of aggregating multi-view geometric and appearance features. The second Transformer (Pixel-align Aggregation) specialized, using attention mechanism translate these fused features into the structured parameters of our semantic 3D Gaussians. To validate this, we trained a baseline with single, monolithic Transformer, and the results, presented in Table 6, confirm our design is superior.
Table 6: Ablation study of our two-stage Transformer architecture. The baseline "w/o Pixel-align
Aggregation" uses single, monolithic Transformer.
Architecture
PSNR 1
SSIM
LPIPS
w/o Pixel-align Aggregation
21.183
0.891
0.067
Full Model (Ours)
23.489
0.916
0.045
26


Reviews:
[{"review_id": "HVhIJIxDOH", "reviewer": "Reviewer_as2P", "summary": "This paper presents a unified framework for 3D human reconstruction, novel view synthesis, and semantic body-part segmentation, from a single image. Built upon 3DGS, the proposed method integrates human appearance prior from SV3D and 2D image features (CLIP image encoder, Dinov2) to achieve semantically consistent 3D representations.  \nThe network consists of two core Transformer modules. The first Tranformer aggregates 3D features and human geometry across source views into pixel-aligned 3D point maps,. The second one applies an attention-based mechanism to project these features into a set of semantic 3D Gaussians, each encapsulating geometry, appearance, and semantic cues. By representing outputs as Gaussians and defining a differentiable rasterizer, the method allows direct rendering of feature maps (including RGB, depth, or segmentation maps) from novel views, supporting a multi-task supervision strategy without the need for per-subject optimization or fine-tuning.   \nExperiments are conducted on 2K2K & THuman2.1 to compare human 3D segmentation & 3D reconstruction. The proposed method achieves better numeric performance than previous methods, e.g. Human3Diffuison, Sapiens, on these benchmarks. In addition, exciting qualitative results are shared.", "questions": "See weakness.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths. \n1. A unified framework.  \nUnlike traditional two-stage approaches, the proposed method, HUMAN3R, offers a single forward-pass solution for 3D reconstruction, semantic segmentation, and novel-view synthesis, all within a Transformer-based pipeline.   \nAlso, the design of two transformer blocks—one for pixel-aligned point regression, and another for semantic 3D Gaussian synthesis—is both modular and generalizable, enabling flexible integration of 2D priors and geometric reasoning.  \n2. Encouraging results.  \nQuantitative and qualitative results on benchmarks and Internet images are quite promising.  \n\nWeakness. \n1. Unclear details.  \nIn experiments section, the paper shows that they use PIXIE as the SMPL paramters estimator. While PIXIE estimates SMPL-X parameters. Different from SMPL, SMPL-X also describe the facial expresion and hand gesture. As shown in Fig. 4, the proposed method achieves better quality at the face and hand part. It would be a surprise if only body parameters are used.   \nAlso, very limited information about the \"In-the-wild Dataset\" they collected for testing, e.g. image source, standard to select the image, how to semgent the human region, etc.   \n2. unvalided designs.  \nThe reason of freezing the 2 image encoders has not been properly explained. So a reasonable guess would be whether train the whole model togather would achieve further improvement.   \n3. Missing qualitative results on evaluation benchmarks.  \nTo further show the superior of the proposed method, qualitative results on evaluation benchmarks would be helpful to directly demonstrate the advance part of the proposed method.", "was_revised": true, "final_justification": "The results look promising and the proposed designs have been validated in the paper and rebuttal. Issues I observed have been properly solved in rebuttal. Therefore, I raise the final score.", "rebuttal": "We thank the reviewer for the insightful and thorough feedback and for acknowledging our core contributions, which introduce a unified and single-pass Transformer framework for joint 3D reconstruction, segmentation, and synthesis. Below, we provide clarifications addressing your concerns.\n\n**W1: The technical details of PIXIE [1]?**\n\nThank you for this sharp observation! The reviewer is correct that HUMAN3R utilizes PIXIE [1] to estimate parameters for the `SMPL-X model`. The high-fidelity results for the hands and face benefit from the expressive priors provided by SMPL-X. We have corrected \"SMPL\" to \"SMPL-X\" in the revision.\n\n**W2: Details of the In-the-wild dataset?**\n\nWe apologize for this omission. Our in-the-wild test set contains 8 images from public CC0 sources, selected to test generalizability under challenging scenarios (e.g., diverse poses and identities). The foreground human masks are automatically generated using the `Segment Anything Model (SAM)`[2] . \nWe will add a subsection to the supplementary materials with these details.\n\n**W3: Why freezing image encoders?**\n\nThank you for this important question. We freeze the image encoders based on DINOv2 [3]'s best practices to leverage its pre-trained features and  to maintain the training efficiency by only training a lightweight decoder. We validated the design choice with an ablation study on the 2K2K dataset, which shows that fine-tuning the image encoder only provides marginal gains (+0.032 PSNR) in our setting.\n\n| Method | PSNR ↑ | SSIM ↑ | LPIPS ↓ |\n| :--- | :---| :--- | :--- | \n| Fine-tuned DINOv2 | 23.521 | 0.920 | 0.046 | \n| Frozen DINOv2  | 23.489 | 0.916 | 0.045 |\n\nWe will incorporate it into the ablation study in the revision.\n\n**W4: Missing qualitative results on evaluation benchmarks.**\n\nWe thank the reviewer for this constructive feedback.  However, according to `NeurIPS 2025 policy, we are prohibited from providing images, videos, or external links during the rebuttal period.`\n\nTo ensure `clarity and transparency`, we summarize our qualitative results: Qualitatively, on the THuman2.1 and 2K2K benchmarks, our method excels at capturing  details (e.g, face and clothing), yielding higher-fidelity results compared to the overly smooth or blurry surfaces from baseline methods. We are committed to adding  qualitative comparisons in the revision.\n\n---\n**Reference:**\n\n[1] Feng, Yao, et al. \"Collaborative regression of expressive bodies using moderation.\", 3DV 2021\n\n[2] Kirillov, Alexander, et al. \"Segment anything.\", ICCV 2023\n\n[3] DINOv2: Learning Robust Visual Features without Supervision, TMLR 2024"}, {"review_id": "x25QBS8VNE", "reviewer": "Reviewer_dwcT", "summary": "This paper proposes Human3R, a unified network to jointly reconstruct 3D human and the corresponding semantics from a single-view image input. The key idea is to use the SV3D framework to generate multiview images with a number of priors such as SMPL, side normals, etc. and lift them into 3D Gaussians. At the same time, it distills DINOv2 feature to the 3DGS, which contributes to the following segmentation. It was trained on THuman2.1, 2K2K, and Human MVImageNet datasets.", "questions": "See weaknesses.", "limitations": "Yes.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n* The visualization looks promising, both reconstruction and segmentation\n* Open-sourcing the trained model would be beneficial to the community\n* From Tab.4, the proposed Pixel-align Aggregation contributes a lot to the performance. The approach could be helpful in many related 3D tasks. \n\nWeaknesses:\n* The results lack failure cases for in-the-wild settings. I'm curious when and how it fails for in-the-wild images. \n* Lack of comparison with general 3D generation, which shows strong ability in humans as well, e.g. Hunyuan3D, SPAR3D\n\nMinor: \n* Less clear presentation of the contribution. Since each step is not new to the field, what makes your method different the priors are not very well-highlighted in the writing. I'm sure there's a lot of original work based on the results provided, but please state them clearly. \n* Segmentation should be evaluated in third-party datasets, as the methods compared with are general.", "was_revised": true, "final_justification": "The authors addressed my questions during the rebuttal. I lean towards accepting this submission. The idea of jointly conducting reconstruction and segmentation is valuable. I recommend revising the segmentation evaluation to make the conditions, pros, and cons of each method clearer, and incorporating the rebuttal discussion into the final version.", "rebuttal": "We thank the reviewer for their positive feedback and for recognizing the qualitative results and the effectiveness of our `Pixel-align Aggregation module`. We appreciate the opportunity to contribute to the community and we have provided detailed clarifications and responses below to address your concerns.\n\n**W1: Lack of discussion on failure cases.**\n\nWe thank the reviewer for this suggestion! While HUMAN3R demonstrates robust generalization across various challenging scenarios, the model's performance may degrade under two primary conditions:\n\n- Extreme Poses: For extreme poses that are challenging for pose estimation models [1], our model may generate minor misalignments in fine details (e.g., in the arms). However, the overall limb structure remains intact, a robustness afforded by the geometric prior.\n\n- Low-Quality Inputs: When processing low-resolution or heavily compressed images, the input may lack the necessary high-frequency information for high-fidelity reconstruction. It leads to a degradation of fine-grained details, such as facial features or clothing textures in the outputs.\n\nWe will include this in the \"Limitations and Failure Cases\" section. Thank you for your valuable feedback!\n\n\n**W2: Comparison with general 3D generation models.**\n\nWe thank the reviewer for their constructive feedback!\n\nFirst, we would like to clarify that our method was compared against recent 3D generation models, including LGM, GRM, and InstantMesh (which is the base model for Hunyuan3D 1.0). To ensure a fair comparison, we fine-tuned both the LGM and GRM models on our dataset.\n\nFurthermore, to fully address your concern, we have benchmarked our method against Stability AI's recent model, SPAR3D, on the test set of the 2K2K dataset. The results below further demonstrate the effectiveness of our approach.\n\n| Method | PSNR ↑ | SSIM ↑ | LPIPS ↓ |\n| :--- | :---: | :---: | :---: |\n| SPAR3D | 20.295 | 0.854 | 0.063 |\n| Ours | 23.489 | 0.916 | 0.045 |\n\nWe will incorporate these new results into the revised version of the paper. Thank you for your valuable suggestion.\n\n**W3: Clarify the contribution.**\n\nWe appreciate the opportunity to clarify our motivation and contributions.\n\nOur primary insight and contribution is beyond it, as an unified pipeline `synergizing 3D reconstruction and semantic segmentation`. This proposed philosophy can unlock many novel and practical downstream applications, such as semantic-guided 3D reasoning, context-aware human behavior analysis, and interactive semantic editing in immersive AR/VR environments, `which have not yet been thoroughly explored`. To achieve this goal, we first devote substantial effort to `curating data-label pairs`, recognizing the extreme scarcity of labeled 3D datasets. Furthermore, we are pleased to observe the following advantageous properties emerging from our unique design:\n  \n1.  **$\\color{#8B2E22}{\\textbf{Segmentation can benefit reconstruction}}$:** We leverage `Pixel-Align Aggregation` to enhance 3D reconstruction by incorporating semantic information on the 2K2K Dataset. Specifically, we utilize two types of semantic information: (1) pre-trained DINOv2 features and (2) manually annotated semantic maps. Both have been shown to improve reconstruction quality, as semantic segmentation helps the model prioritize specific parts and clothing regions.\n\n| Method | DINOv2  | Semantic Guidance | PSNR ↑ | SSIM ↑ | LPIPS ↓ |\n| :--- | :---: | :---: | :---: | :---: | :---: |\n| Base Model (`w/o Pixel-Align Aggregation`)| ❌ | ❌ | 21.183 | 0.891 | 0.067 |\n| Feature Distillation | ✅ |  ❌ | 22.464 | 0.896 |  0.055 |\n| Ours (Full Model) | ✅ | ✅ | 23.489 | 0.916 | 0.045 |\n\n\n2.  **$\\color{#8B2E22}{\\textbf{Reconstruction  can benefit segmentation}}$:** Limited segmentation data often struggles to achieve convergence for 3D consistency segmentation maps. To address this, we naturally incorporate reconstruction priors, which significantly enhance 3D consistency and improve mIOU from `0.724` to `0.840` **(+16%)**. Notably, on human mesh datasets, our approach achieves better `3D consistency and coherence` compared to 2D foundation models (e.g., Sapiens).\n\nAdditionally, our ablation studies validate the impact of key design components—such as loss functions and geometry guidance. We will release the entire pipeline (code and constructed dataset) to support community adoption.\n  \n**W4: Why not evaluate the 3D segmentation tasks on third-party datasets?**\n\nWe agree that evaluation of diverse datasets is important. However, to the best of our knowledge, our work was the first work to explore a large-scale,  `3D human dataset with detailed, multi-class part-segmentation labels (28 classes)`. This is precisely why we invested significant effort in curating our dataset. \nWe believe our released dataset will serve as a valuable new benchmark for the community and help drive future research in this domain.\n\n---\n**Reference:** \n\n[1] Feng, Yao, et al. \"Collaborative regression of expressive bodies using moderation.\", 3DV 2021\n\n[2] SPAR3D: Stable Point-Aware Reconstruction of 3D Objects from Single Images, arXiv 2025\n\n[3] SiFU:Side-view Conditioned Implicit Function for Real-world Usable Clothed Human Reconstruction, CVPR 2024\n\n[4] HumanSplat: Generalizable Single-Image Human Gaussian Splatting with Structure Priors, NeurIPS 2024\n\n[5] PSHuman: Photorealistic Single-image 3D Human Reconstruction using Cross-Scale Multiview Diffusion and Explicit Remeshing, CVPR 2025"}, {"review_id": "7eMhhEgwVv", "reviewer": "Reviewer_wsxq", "summary": "This paper proposes a novel feed-forward end-to-end pipeline to reconstruct human geometry, appearance, and semantic features from a single image, using 3DGS as the representation, allowing efficient reconstruction and inference and many downstream applications like AR/VR.", "questions": "1. I don't understand why the authors introduce two transformer modules to predict the 3DGS from multi-view images (if I understand correctly, one for feature aggregation and another for predicting the Gaussians) instead of directly using a single module/transformer. Could the authors elaborate more on the benefits of such a design?\n2. While training an end-to-end reconstruction model is efficient, such a data-driven approach may lack generalizability and result in poorer reconstruction quality. I wonder if the authors could elaborate on why they believe an end-to-end model is preferable to optimization-based methods. For example, they could provide comparison results with those obtained by directly optimizing a 3DGS from multi-view images generated by the diffusion model and the per-image semantic feature fields.\n3. How is the SMPL model used in the proposed method? Is it provided as input or estimated from some human mesh recovery model?", "limitations": "The authors mention using web-scale human datasets to train the model in their future work, which may raise potential ethical concerns.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The idea of end-to-end reconstruction is promising, with good performance and qualitative results.\n2. Monocular reconstruction from a single image is still an open-ended and highly challenging task, especially for humans. It is beneficial to the community to push this field forward with new methods.\n\nWeaknesses:\n1. Unclear Motivation: In the introduction, the authors primarily discuss the limitations of existing approaches that utilize monocular videos or sparse multi-view inputs. However, the paper fails to provide a convincing motivation for using monocular images as inputs instead of addressing these limitations. I believe monocular reconstruction from a single image is a highly ill-posed problem and could be even harder to solve.\n2. Rely on a 2D diffusion model. I believe the most crucial component of the proposed method is the pre-trained diffusion model, which can generate 3D consistent multi-view images from a monocular input and addresses the ill-posed 2D-to-3D problem. Therefore, the performance of the method would largely depend on the quality of the diffusion model, which is not this paper's contribution.\n3. Lack of novelty: The paper proposes a pipeline to first use 3D priors to lift the input 2D image to 3D, but each module in the proposed pipeline either directly uses a pretrained model (e.g, the 2D diffusion model) or follows existing works (e.g., feed forward pixel-aligned 3DGS) without too much specific model design or motivation on why such design of combing existing techniques would be helpful to the final result. This makes the technical contribution weak.", "was_revised": true, "final_justification": "After the rebuttal, the authors addressed most of my concerns, especially the motivation and the significance of the proposed method. Although the writing of the current version of the paper still lacks clarity, I appreciate the value of jointly learning reconstruction and semantics claimed in this paper and incline towards acceptance.  I suggest the authors updating the introduction to clarify the motivation of this work and the method section to make the key technical contribution (i.e the feature aggregation transformer and the pixel-align aggregation transformer) clearer.", "rebuttal": "We thank Reviewer wsxq for the critical and detailed feedback, which has prompted us to conduct several new experiments to validate our core claims.\n\n**Q1: Why task monocular images as inputs ?**\n\nThank you for this important question! We agree that monocular 3D reconstruction is a highly ill-posed and challenging problem. However, we believe its difficulty is precisely what makes it a fundamental and valuable area of research, for two main reasons:\n\n1. Greater Flexibility and Efficiency: Inspired by HumanSplat and PShuman [1,2], a single-image to 3D pipeline offers significant flexibility. It allows users to reconstruct a full 3D representation from just one picture, which is essential for many downstream applications in fields like gaming, movies, fashion, and AR/VR, where multi-view data is often unavailable.\n  \n2. A Unified and Extensible Framework: The architecture is modular, allowing us to adapt it for sparse or multi-view inputs by replacing specific components (e.g., using a multi-view spml estimation model). In fact, our model can be extended to handle two input views by simply replacing the generated latent  with those from the GT input latents (encode input images via a VAE model), as shown in our experiments (Tables 1 and 3). This shows that our single-view approach provides a adaptable foundation for sparse input scenarios.\n\n\n**Q2:  Reliance on Diffusion and Lack of Novelty.**\n\nYour insight regarding the technical contribution is enlightening. We appreciate the opportunity to clarify our motivation and contributions.\n\nWe group these points as they relate to the core premise and contribution of our work. We respectfully argue that our work presents significant novelty beyond a simple aggregation of techniques. The core innovation lies in the synergistic integration and non-trivial adaptation of existing components to solve the novel and challenging task of joint, single-image human reconstruction and segmentation.\n\nOur motivation is to tackle the highly ill-posed but practical single-image problem, which is now feasible due to powerful 2D generative priors. Our contribution is not the prior itself, but the framework that harnesses and lifts these 2D features into a coherent, semantic 3D representation.\n\nWe appreciate the opportunity to clarify our motivation and contributions. Our primary insight and contribution is beyond it, as an unified pipeline `synergizing 3D reconstruction and semantic segmentation`. This proposed philosophy can unlock many novel and practical downstream applications, such as semantic-guided 3D reasoning, context-aware human behavior analysis, and interactive semantic editing in immersive AR/VR environments, `which have not yet been thoroughly explored`. To achieve this goal, we first devote substantial effort to `curating data-label pairs`, recognizing the extreme scarcity of labeled 3D datasets. Furthermore, we are pleased to observe the following advantageous properties emerging from our unique design:\n  \n1.  **$\\color{#8B2E22}{\\textbf{Segmentation can benefit reconstruction}}$:** We leverage `Pixel-Align Aggregation` to enhance 3D reconstruction by incorporating semantic information on the 2K2K Dataset. Specifically, we utilize two types of semantic information: (1) pre-trained DINOv2 features and (2) manually annotated semantic maps. Both have been shown to improve reconstruction quality, as semantic segmentation helps the model prioritize specific parts and clothing regions.\n\n| Method | DINOv2  | Semantic Guidance | PSNR ↑ | SSIM ↑ | LPIPS ↓ |\n| :--- | :---: | :---: | :---: | :---: | :---: |\n| Base Model (`w/o Pixel-Align Aggregation`)| ❌ | ❌ | 21.183 | 0.891 | 0.067 |\n| Feature Distillation | ✅ |  ❌ | 22.464 | 0.896 |  0.055 |\n| Ours (Full Model) | ✅ | ✅ | 23.489 | 0.916 | 0.045 |\n\n\n2.  **$\\color{#8B2E22}{\\textbf{Reconstruction  can benefit segmentation}}$:** Limited segmentation data often struggles to achieve convergence for 3D consistency segmentation maps. To address this, we naturally incorporate reconstruction priors, which significantly enhance 3D consistency and improve mIOU from `0.724` to `0.840` **(+16%)**. Notably, on human mesh datasets, our approach achieves better `3D consistency and coherence` compared to 2D foundation models (e.g., Sapiens).\n\nAdditionally, our ablation studies validate the impact of key design components—such as loss functions and geometry guidance. We will release the entire pipeline (code and constructed dataset) to support community adoption.\n\n\n**Q3:  Why two Transformers instead of a single transformer?**\n\nThis is an excellent question. The first Transformer (Feature Aggregation)  focuses on the general task of aggregating multi-view geometric and appearance features. The second Transformer (Pixel-align Aggregation) is specialized, using an attention mechanism to translate these fused features into the structured parameters of our semantic 3D Gaussians. To validate this, we have conducted ablation studies with a single, monolithic Transformer, and the results confirm our design is superior:\n\n| Architecture | PSNR ↑ | SSIM ↑ | LPIPS ↓ |\n| :--- | :---: | :---: |:---: |\n| w/o Feature Aggregation Transformer | 22.203 | 0.890 | 0.064|\n| w/o Pixel-align Aggregation Transformer | 21.183 | 0.891 | 0.067|\n| Full Model (Ours) | 23.489 | 0.916 | 0.045 |\n\n\n**Q4: Why an end-to-end model instead of per-subject optimization?**\n\nThank you for your constructive feedback. We agree that optimization-based methods can achieve more high-quality results. However, we respectfully offer a different perspective on the primary goal of our work.\n\n1. Unlike optimization-based methods that often require minutes to hours per subject, our model is designed for efficiency and scalability , enabling 3D reconstruction in a matter of seconds. We believe this is a critical feature for many real-world applications where speed is essential.\n\n2. Furthermore, the two approaches are not mutually exclusive. As explored in recent work like InstantSplat [3], efficient, end-to-end models can serve as an excellent initialization for optimization-based methods, significantly accelerating their convergence. This potential synergy further highlights that developing fast, feed-forward models is a valuable research direction.\n\n\n**Q5: How is the SMPL model used?**\n\nThank you for this important question. For an input image, we first estimate SMPL(-X) parameters with PIXIE [4]. The resulting mesh provides a structural guidance for the human's pose and shape. \n\n**Q6: How is the SMPL model used?**\nWe appreciate your important point.  Our dataset will be built exclusively from permissively-licensed  images (e.g., HGS-1M [5]). For any future work, we will adhere to strict ethical guidelines, including data source transparency and privacy filtering.\n\n---\n**Reference:**\n\n[1] HumanSplat: Generalizable Single-Image Human Gaussian Splatting with Structure Priors, NeurIPS 2024\n\n[2] PSHuman: Photorealistic Single-image 3D Human Reconstruction using Cross-Scale Multiview Diffusion and Explicit Remeshing, CVPR 2025\n\n[3] InstantSplat: Sparse-view Gaussian Splatting in Seconds, Arxiv 2024\n\n[4] Feng, Yao, et al. \"Collaborative regression of expressive bodies using moderation.\", 3DV 2021\n\n[5] SIGMAN: Scaling 3D Human Gaussian Generation with Millions of Assets, ICCV 2025"}, {"review_id": "mT9QoxPcyq", "reviewer": "Reviewer_GaxS", "summary": "This paper introduces Human3R, a method that performs feed-forward human reconstruction and human-part semantic segmentation from a single image. The main components of the approach are: (1) Given a single image, a diffusion model performs novel view synthesis; (2) Using an estimated 3D human mesh as guidance and camera Plucker embeddings, pose-conditioned images are created; (3) A Transformer can then aggregate those multi-view pose-conditioned images and unproject them to the 3D Gaussians; (4) From the input image, some features are extracted using DINOv2. Those features are linearly combined by leveraging the attention weights from the previous network. (5) Finally, the rendering can be done for colors and semantics using a rendering function with shared weights.\n\nExperiments show that this approach obtains SOTA results in both 3D segmentation and reconstruction.", "questions": "I would like to hear from the authors on the following points:\n- **Novelty:** What are the main contributions and insights of this paper? From my understanding, it looks like the proposed approach is mainly an aggregation of techniques proposed in prior works.\n- **Technical detail** How are the depth maps and offsets predicted in the feature aggregation component?\n- **Evaluation:** How can we perform 3D segmentation with Sapiens (2 input images)? How can Human3R handle more than one image?\n- **Ablations:** Could models in the spirit of DUSt3R provide a better initialization?", "limitations": "Yes.", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "# Strengths\n\nThe proposed approach successfully extends 3DGS Feature Field to humans. Even if the idea of adding feature prediction is not new, applying such a method to humans is not straightforward.\n\nHuman3R obtains SOTA performance in both 3D human segmentation and reconstruction.\n\nThe authors will release the code and the introduced dataset.\n\n\n# Weaknesses\n\nThe overall writing of the paper could be polished (unclosed parenthesis L68, \"Our approach is builds upon\", missing capital letter L251, inconsistent sentences such as L243, ...).\n\nI feel like the method is an aggregation of many \"recipes\" that worked in other works. Specifically, multi-view images are generated using SV3D [65], and the SMPL guidance with side-view normal image follows works like Sifu [69]. The cross-view attention module does not seem different from prior large reconstruction models. The use of DINOv2 features and their aggregation also seems very close to LUDVIG [26]. The training loss is in the same spirit as prior works such as LSM [27]. I do not argue that the paper is not novel at all as elements come from many different papers, and the application is new, but I wonder what insights this paper gives, and how the community can build on top of it.\n\nAdditionally, I have the following doubts (these points could probably be clarified):\n- L143: How are the depth maps and offsets predicted?\n- Table 1: How is the Sapiens [15] baseline computed? I understand how it can work when combined with Human3Diffusion [47], but I struggle to understand how it could alone predict the 3D segmentation given 2 images.\n- Table 1: How can Human3R support 2 input views?\n- L265: If the ability to establish correspondences, why not use recent models like DUSt3R [a], VGGT [b], or even DUNE [c] (that is also capable of predicting 3D human meshes)?\n\n\n\n[a] Wang, Shuzhe, et al. \"Dust3r: Geometric 3d vision made easy.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\\\n[b] Wang, Jianyuan, et al. \"Vggt: Visual geometry grounded transformer.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.\\\n[c] Sarıyıldız, Mert Bülent, et al. \"DUNE: Distilling a Universal Encoder from Heterogeneous 2D and 3D Teachers.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.\n\n# Overall recommendation\n\nThis paper presents an approach that performs human reconstruction and 3D segmentation. The authors leverage the latest advances in 3DGS that add a feature field and leverage a 2D foundation model, which benefits both tasks. Despite those strengths, I feel like the writing could be polished, and I wonder if the method is technically novel enough for NeurIPS. Still, I rate this paper as \"Borderline Accept\" as it is mostly clear, the application of 3DGS Feature Field is novel in the human field, and the proposed approach obtains good results.", "was_revised": true, "final_justification": "After reviewing the author's rebuttal, I no longer have concerns related to a lack of novelty or insights. Additionally, answers to my technical questions allowed me to understand the method and baselines better. Weaknesses highlighted by other reviewers were also addressed during the rebuttal.\n\nI think this paper presents a nice research idea (using the synergy between 3D reconstruction and segmentation); the method relies on the latest advances in the field of 3DGS features fields and human reconstruction, and is clearly explained. Comparisons to the SOTA approaches and ablation studies validate the approach on multiple widely used datasets. I increase my final rating to \"Accept\".", "rebuttal": "We sincerely thank the reviewer for the positive feedback and insightful summary. We appreciate the recognition of our work's key strengths: the non-trivial extension of 3DGS Feature Fields to humans, the state-of-the-art performance achieved, and our commitment to releasing the code and dataset to foster future research in the community. We sincerely hope our responses below adequately address your concerns.\n\n**Q1: Polish Writing.**\n\nThank you for your careful and detailed review. We will correct all typos and inconsistencies in the revised manuscript, including the specific points you raised:\n- L68: The missing parenthesis \")\" will be corrected.\n- L252: The capitalization of \"We\" will be fixed.\n- L243: The inconsistent sentence will be rewritten for clarity.\nFurthermore, we will conduct a thorough proofread of the entire paper to improve its overall polish.\n\n**Q2: Clarification of the Novelty**\n\nWe appreciate the opportunity to clarify our motivation and contributions.\nOur primary insight and contribution is beyond it, as an unified pipeline `synergizing 3D reconstruction and semantic segmentation`. This proposed philosophy can unlock many novel and practical downstream applications, such as semantic-guided 3D reasoning, context-aware human behavior analysis, and interactive semantic editing in immersive AR/VR environments, `which have not yet been thoroughly explored`. To achieve this goal, we first devote substantial effort to `curating data-label pairs`, recognizing the extreme scarcity of labeled 3D datasets. Furthermore, we are pleased to observe the following advantageous properties emerging from our unique design:\n  \n1.  **$\\color{#8B2E22}{\\textbf{Segmentation can benefit reconstruction}}$:** We leverage `Pixel-Align Aggregation` to enhance 3D reconstruction by incorporating semantic information on the 2K2K Dataset. Specifically, we utilize two types of semantic information: (1) pre-trained DINOv2 features and (2) manually annotated semantic maps. Both have been shown to improve reconstruction quality, as semantic segmentation helps the model prioritize specific parts and clothing regions.\n\n| Method | DINOv2  | Semantic Guidance | PSNR ↑ | SSIM ↑ | LPIPS ↓ |\n| :--- | :---: | :---: | :---: | :---: | :---: |\n| Base Model (`w/o Pixel-Align Aggregation`)| ❌ | ❌ | 21.183 | 0.891 | 0.067 |\n| Feature Distillation | ✅ |  ❌ | 22.464 | 0.896 |  0.055 |\n| Ours (Full Model) | ✅ | ✅ | 23.489 | 0.916 | 0.045 |\n\n\n2.  **$\\color{#8B2E22}{\\textbf{Reconstruction  can benefit segmentation}}$:** Limited segmentation data often struggles to achieve convergence for 3D consistency segmentation maps. To address this, we naturally incorporate reconstruction priors, which significantly enhance 3D consistency and improve mIOU from `0.724` to `0.840` **(+16%)**. Notably, on human mesh datasets, our approach achieves better `3D consistency and coherence` compared to 2D foundation models (e.g., Sapiens).\n\nAdditionally, our ablation studies validate the impact of key design components—such as loss functions and geometry guidance. We will release the entire pipeline (code and constructed dataset) to support community adoption.\n\n**Q3: Technical Details (Depth/Offset Prediction, Sapiens/Human3R in Table 1).**\n\n- Depth/Offset Prediction: Inspired by Splatter Image [1], the depth maps and 2D offsets are predicted by a small, two-branch MLP head attached to the output of our first (feature aggregation) Transformer. This head regresses per-pixel depth and offset values, which are then used to unproject pixels into an initial 3D XYZ.\n- Sapiens + Human3Diffusion (2-view baseline): To create a strong baseline for two-view 3D segmentation, we adapted the official open-source implementation of Human3Diffusion. Since their framework already uses a multi-view diffusion model and a 3D Gaussian Splatting representation similar to ours, we \"hacked\" their 3DGS renderer to support feature rendering. This allowed us to establish a baseline with a comparable architecture. As shown in Table 1, Human3R achieves **higher reconstruction fidelity with a lower inference time**, demonstrating the effectiveness of our integrated approach.\n- Human3R (2-view support): We simply bypass the diffusion-based view generation and feed the second ground-truth view directly into our feature aggregation module. This highlights our framework's flexibility. We will add these implementation details to the experimental setup section in our revision.\n  \n**Q4: Using newer correspondence models (e.g., DUSt3R).**\n\nThis is an excellent and insightful suggestion. We would like to clarify that our primary baseline, LSM, is already built upon DUSt3R. As mentioned in lines 226-227 of our paper, we fine-tuned it on our human dataset to ensure a fair and direct comparison. \n\nRegarding the newer models you mentioned (VGGT [2], DUNE [3]), they were not yet established or widely adopted at the time we were conducting our primary experiments, and integrating them would require substantial effort to adapt and fine-tune for our specific downstream task. We thank the reviewer for this idea and will add it to our future work/discussion section. \n\n**Q5: Appendices in supplementary material.**\n\nThank you for careful review. We will move all appendices to the supplementary material for the camera-ready version, as per NeurIPS formatting guidelines.\n\n\n---\n**Reference:** \n\n[1] Splatter Image: Ultra-Fast Single-View 3D Reconstruction\n\n[2] VGGT: Visual Geometry Grounded Transformer, CVPR 2025\n\n[3] DUNE: Distilling a Universal Encoder from Heterogeneous 2D and 3D Teachers, CVPR 2025"}]

Author rebuttal:

Meta-review: This submission receives 3 accepts and 1 borderline accept. During the rebuttal discussion period, all reviewers' concerns have been well addressed. As stated by reviewers, monocular reconstruction from a single image is still an open-ended and highly challenging task and this submission proposes a single forward-pass solution utilizing the synergy between 3D reconstruction and segmentation. Experiments have demonstrated the effectiveness of the proposed method. The recommendation is acceptance. And authors are encouraged to include the clarification and update covered in the rebuttal discussion period to the revision.
Decision: Accept (poster)