OccFusion: Rendering Occluded Humans with Generative Diffusion Priors
Authors: Adam Sun
Tiange Xiang
Scott Delp
Li Fei-Fei
Ehsan Adeli
Venue: NeurIPS 2024 poster (2024)
Keywords: vision
gaussian splatting
human
rendering
diffusion
Primary area: machine_vision

Abstract: Existing human rendering methods require every part of the human to be fully visible throughout the input video. However, this assumption does not hold in real-life settings where obstructions are common, resulting in only partial visibility of the human. Considering this, we present OccFusion, an approach that utilizes efficient 3D Gaussian splatting supervised by pretrained 2D diffusion models for efficient and high-fidelity human rendering. We propose a pipeline consisting of three stages. In the Initialization stage, complete human masks are generated from partial visibility masks. In the Optimization stage, 3D human Gaussians are optimized with additional supervisions by Score-Distillation Sampling (SDS) to create a complete geometry of the human. Finally, in the Refinement stage, in-context inpainting is designed to further improve rendering quality on the less observed human body parts. We evaluate OccFusion on ZJU-MoCap and challenging OcMotion sequences and found that it achieves state-of-the-art performance in the rendering of occluded humans.

OccFusion: Rendering Occluded Humans with Generative Diffusion Priors
Adam Sun', Tiange Xiang", Scott Delp, Li Fei-Feit, Ehsan Adelit
Stanford University
(adsun, xtiange)@stannorddd edu
PSNR (2I0 (UU-MMCap)
21
HanasNatr
Thcl soaaa
Coctadllaaa (reo)
Orcifaaien
Training Time (mins)
2o.
10h
OccGauHuman
ZJU-MoCap
OccFusion
OccGauHuman
OcMation
OccFusion
Figure 1: Reconstructing humans from monocular videos frequently fails under occlusion. this paper, introduce OceFusion, method that combines 3D Gaussian splatting with 2D diffusion priors for modeling occluded humans. Our method outperforms the state-of-the-arr rendering quality and efficiency, resulting in clean and complete renderings free of artifacts.
Abstract
Most existing human rendering methods require every part of the human be fully visible throughout the input video. However, this assumption does not hold in real-life settings where obstructions are common, resulting in only partial visibility of the human. Considering this, we present OceFusion, an approach that utilizes efficient 3D Gaussian splatting supervised by pretrained 2D diffusion models for efficient and high-fidelity human rendering. We propose pipeline consisting of three stages. the Initialization stage, complete human masks are generated from partial visibility masks. In the Optimization stage, human 3D Gaussians are optimized with additional supervision by Score-Distillation Sampling (SDS) to create complete geometry of the human. Finally, in the Refinement stage, in-context inpainting is designed to further improve rendering quality the less observed human body parts. We evaluate OccFusion on ZJU- MoCap and challenging OcMotion sequences and find that achieves state-of- the-art performance in the rendering of occluded humans. Project page: https: //cs. sttss.saafoon ord. wwweedtiiinnpeeeeeeeea wttnnn////oojccs/ccccuueee
Introduction
Rendering 3D humans from monocular in-the-wild videos has been persistent challenge, with significant implications in virtual/augmennted reality, healthcare, and sports. Given a video of a human
"Equal contribution; junior author listed first.
1 Equal mentorship.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

moving around a scene, this task involves reconstructing the appearance and geometry of the human, allowing for the rendering of the human from novel views.
When faced with the problem of human reconstruction from monocular video, several works based 3DD Gaussian
on neural radiance fields (NeRFs) have achieved promising results [37 57
19
2
splatting [24] further improves upon NeRF-based rendering methods for better performance. By representing the human not as an implicit radiance field but as set of explicit 3D Gaussians, methods like GauHuman [4] and 3DGS-Avatar Avatar 4 are able to render humans comparable in quality to NeRF methods, while taking only a few minutes to train and less than a second to render.
While most human rendering studies assume clean, unobstructed environments, real-world settings like hospitals, stadiums, and construction sites involve frequent occlusions. Current methods struggle in these conditions, often producing artifacts, floating elements, or incomplete body parts. Solutions like OccNeRF (64) and Wild2Avatar [63] attempt address occlusions but are limited by high computational demands and lengthy training times, making them impractical and restricting their real-world applicability.
In this work, we introduce OccFusion, an efficient yet high quality method for rendering occluded humans. To gain improved training and rendering speed, OccFusion represents the human as set of 3D Gaussians. Like almost all other human rendering methods [64 63 67 OceFusion assumes accurate priors such as human segmentation masks and poses are provided for each frame, which can be obtained with state-of of-toe--tt-t off- shelhh estimators such as SAM [25] and HMR 2.0 [8]. However, to ensure complete and high-quality renderings under occlusion, OccFusion proposes to utilize generative diffusion priors, more specifically pose-conditioned Stable Diffusion 1.5 [88 with
ControlNet (76) plugins, to aid in the reconstruction process.
Our approach consists of three stages: (1) The Initialization Stage: we utilize segmentation and pose priors to inpaint occluded human visibility masks into complete human occupancy masks to supervise later stages. (2) The Optimization Stage: we initialize set of 3D Gaussians and optimize them based on observed regions of the human, applying consiiondtt help ensure completeness of the modeled human body in both the posed and canonical space. (3) The Refinement Stage: we utilize pretrained generative models to inpaint unobserved regions of the human with context from partial observations and renderings from the previous stage, further improving the quality of the renderings. Despite taking only 10 minutes to train, our method outperforms the state-of-the-ar in rendering humans from occluded videos.
Sampling (SDS) to
In summary, our contributions are: (i) We propose OccFusion, the first method to combine Gaussian splatting with diffusion priors for the rendering of occluded humans from monocular videos. Mul- tiple novel components are proposed along with three-stage pipeline consisting of Initialization, Optimization, and Refinement stages. (ūi) We demonstrate that OceFusion achieves state-of ahe-art efficiency and rendering quality of occluded humans both simulated and real-world occlusions.
2
Related Work
2.1 Neural Human Rendering
Traditional methods to reconstruct humans usually require dense arrays of cameras [0. or
4 5. both of which are unobtainable for in-the-wild scenes. To solve this problem, Neural Radiance Fields (NeRFs) [37] have recently been used to model dynamic humans from monocular videos [57 9 19
depth information 7 50
17
72, 52 These methods achieve high-quality novel
view synthesis by parametrizing the human body using an SMPL (35) pose prior and modeling it as radiance field. However, since NeRFs depend on large Multi-Layer Perceptrons (MLPs), they are computationally expensive, usually taking days to train and minutes to render [4, 46). To
speed up NeRF-based models, multi-resolution hash encoding 44
43 6 [8] and generalizability
have been proposed. However, these methods either face a rendering bottleneck [4]]]] or expensive pre-training process, both of which affect their efficiency.
[1 2 1 27
Point-based rendering methods like 3D Gaussian splatting (2 greatly accelerate the rendering of static and dynamic scenes. Recently, there have been an abundance of works applying 3D Gaussian
splatting to human rendering tasks [46] 14 26
20 29 28
42 12] Like NeRF-
based approaches, Gaussian splatting- approaches represent the human in canonical space

---

and use Linear Blend Skinning (LBS) to transform the human into the posed space. Gaussian splatting methods achieve sfte-offtheearr performance dynamic humans with fast training times and real-time rendering, causing them to be the more desired method 46 14
2.2 Occluded Human Rendering
Reconstructing complex scenes in the wild a well-studied problem. NeRF-W W [6 and other works 3.47 [ are able to account for photometric variation and transient occluders, allowing them to render consistent representations from unconstrained image collections. However, these works are not designed to handle dynamic objects like humans.
Rendering humans in occluded settings, on the other hand, relatively understudied. Sun er al. [51] utilize a layer-wise scene decoupling model to decouple humans from occluding objects. OccNeRF (64] combines geometric and visibility priors with surface-based rendering train human NeRF model, while Wild2Avatar [63] proposes an occlusion-awas scene parametrization scheme to decouple the human from the background and occlusions. While these works provide decent renderings of humans free of occlusions, they are slow and impractical due to their usage of NeRFs. concurrent work to ours OceGaussian [67], which also proposes to model occluded humans with 3D Gaussians by performing occlusion feature query in occluded regions. We provide comparisons to their published results in Table
Generative Diffusion Priors
Inferring the appearance of unobserved regions of 3D scenes requires the usage of generative models. The recent success 2D diffusion models has made them the preferred model to use for generation
54
48 32
To lift 2D diffusion models for 3D content generation, DreamFusion 45
proposed Score Distillation Sampling (SDS), a commonly used method for utilizing pre-trained 2D
diffusion model to supervise 3D content generation
30
70
53 55).
Diffusion models can also be used priors for training NeRFs and Gaussian splatting, combining ReconFusion [6] uses SDS in conjunction
reconstruction with generation [6] 78 79 62
66
73
with multi-vit -iew conditioning to synthesize the appearance of unobserved regions scene from sparse views, while BAGS 78 utilizes SDS to supervise Gaussian splatting model.
3 Preliminaries
Before introducing method, we provide overview of key fundamentals in 3D human modeling using SMPL (subsection 3.II). Then, we discuss 3D Gaussian splatting, and how it can be applied to human modeling (subsection 3.2) Finally, propose OcoGauHuman, simple improvement of GauHuman [4] that is better designed for occluded human rendering (subsection 3.3).
3.1
3D Human Modeling
SMPL 35 is model that parametrizes the human body with 3D surface mesh. To transform between the canonical space to a pose space, the Linear Blend Skinning (LBS) algorithm is used. Given 3D point x. in the canonical space and the shape 8 and pose parameters of the human, point in the posed space can be calculated as:
K
Xp
(G.(J,0)x (J,0,)))
where J contains K joint locations, G3 and by: are the transformation matrix and translation vector, and Wh E [0, 1] are a set of skinning weights. The SMPI. representation is commonly used
geometric prior for human rendering 664 63 46
14
57 9 72).
3.2 Human Rendering with 3D Gaussian Splatting
3D Gaussian splatting. Gaussian splatting 24
models a scene as set of 3D Gaussians Each Gaussian is defined by its 3D location pj, opacity E [0, 1], center μ4, covariance matrix E,, and

---

spherical harmonic coefficients. The i-th Gaussian is defined #(p=<(p) (p-pa)) During rendering, these 3D Gaussians are mapped from the 3D world space and projected to the 2D image space via a-blending, with the color of each pixel being calculated across the N 3D Gaussians as:
N
C
Σ 2=1
II
(1 ag),,
(2)
where oj the color and a the z-depth ordered opacity. During the training process, 3D Gaussians are adaptively controlled via densification (splitting and cloning) and pruning until they achieve the optimal density to adequately represent the scene.
GauHuman 14] In the line of work that uses 3D Gaussian splatting for human rendering [46]
26 29 [2]. GauHuman is a representative approach due to its balance between efficiency and rendering quality. After initializing 3D Gaussians on the vertices of the SMPL mesh, GauHuman learns representation of the human in canonical space and utilizes LBS to transform each individual Gaussian into the posed space. A pose refinement module M MMPP yuse and an LBS weight field module M ML are used to learn the LBS transformation, and a merge operation based on KL divergence used along with splitting, cloning, and pruning help the 3D Gaussians reach convergence.
We base our method on GauHuman due to its fast training and state-of-the-ar representative ability. GauHuman's 's code is distributed under the S-Lab license and can be accessed here.
3.3 OceGauHuman: An Improved Baseline for Occlusion Handling
In common human rendering tasks, videos are captured in a clean environment, with every pixel in the image belonging to either the human or the background. By using semantic segmentation model such as SAM [25] to preprocess a video, we can train the human rendering model only on pixels labeled as "human" However, occlusions in the videos may lead to sparse observations of the human. As result, fitting NeRF-based human rendering models on only the visible human pixels results in an incomplete geometry with lots of artifacts 66 63
Gaussian ppattting-based rendering models due their explicit geometry and point-based representation. In this section, present three straightforward tweaks of GauHuman [4] to make it perform better on videos with occlusions: (I) Firstly, as discussed above, we train the model on visible human pixels only, ensuring that occlusions do not result in learned sparsity on the human model. (2) We adjust the loss weights to put more weight the mask loss computed between rendered human occupancy maps and the segmentation masks- - we found that this helps learn more crisp human boundaries.
are especially suitable for human modeling tasks
(3) We disable the densification and pruning of 3D Gaussians during training this helps maintain a rather complete human geometry based on the SMPL initialization.
The resulting OccGauHuman model serves as improved baseline for occluded human reconstruc- tion and as a starting point for our method. Benefits brought by our updates compared to the original GauHuman are presented in Table T as well in Figure] 7
4 OceFusion
In our approach, we train a Gaussian splatting-basee human rendering model on the visible pixels of human. However, recovering occluded content for dynamically moving human is not trivial humans are usually in challenging poses, and complex occlusions can cause additional issues. It also essential to preserve a consistent human appearance and geometry across different frames. Considering these challenges, we propose our method OccFusion in multiple separate stages. In the Initialization stage (section 4.I). we inpaint occluded binary human masks for more reliable geometric guidance. In the Optimization stage (section 422) we use the inpainted masks to train a human rendering model based on GauHuman [1] while using Score Distillation Sampling (SDS) constraints on both the posed space and canonical space. In the Refinement stage (section 4.3). we fine-tune the trained model from the Optimization Stage with in-context inpainting to further refine the appearance of the human. An overview of our OccFusion is shown in Figure 2

---

Diflusion
Upcaas//uccwardd
(C)
(M)
II
(1)
>
>
Occluded human
Init Stoge Sec. 4.1
Optim Stage Soc. 4.2
Refine Stage Sec. 4.3
Rendered human
Figure 2: OceFusion achieves occluded human rendering via three sequential stages. the Initial- ization Stage, we recover complete binary human masks {M} from occluded partial observations {T) with the help of segmentation priors {M) and pose priors (P). (M) will be further used help optimize the 3D Gaussians II in subsequent stages. In the Optimization Stage, apply [P) conditioned SDS on both posed human and canonical human to enforce the human occupancy to remain complete. In the Refinement Stage, we use the coarse human renderings { from the Optimization Stage to help generate missing RGB values in {I} through our proposed in-context inpainting. Through this process, both the appearance and geometry of the human are fine-tuned to be high fidelity. Training of all three stages takes only 10 minutes on single Titan RTX GPU.
4.1 Initialization Stage: Recovering Human Geometry from Partial Observations
Generative diffusion models 48 have demonstrated promise to be used as priors for different tasks (2 [33.
The most straightforward method is to utilize a precomputed segmentation prior M and pose prior P to condition & [39 [76 to inpaint - M - the image regions that are not occupied by the human. However, there are two significant barriers such a straightforward approach.
Occluded human
20 pose
Pose conditioned SD generations
Simplified 20 pose
Pose conditioned SD generations
Figure Stable Diffusion 1.5 generations 48 conditioned on challenging pose P. While
conditioning the original pose results in multiple limbs and other abnormalities, our method of simplifying pose by removing self-occluded joints results more feasible generations.
Conditioned human generation cannot handle challenging poses. It is true that conditioned diffusion prior & is able to generate detailed images while staying consistent with the condition. However, since diffusion models like & are usually overfitted more commonly seen poses, O usually fails to generate reasonable images when conditioned on challenging poses (see Figure middle column). We attribute this limitation the inappropriate 2D representation of P when joints occlude each other, is impossible to tell which joints are closest to the camera when they are projected to 2D. So, we propose to simplify the 2D representation of P. We apply Z-buffer test on the depth map rendered from the SMPL mesh 35 and then calculate the distance d between its z-axis location and the corresponding 2D z-buffer. Given pre-defined threshold a, we deem a joint self-occluded if d > a. Self-occluded joints are ignored when projecting 3D joints onto the 2D) canvas for conditioning & (see Figure rright column). Our simplification improves the generation quality of & for challenging poses.
Per-frame inpainting cannot guarantee cross-frame consistency, Compared to image generation models, video generation models [1]
60 7 are less accessible and much more expensive to run. Without an explicit modeling of object motion the video, frame-by- generation with an image generative model leads cross-free fras-frame inconsistency, which is not desirable for human reconstruction

---

1L
Inpainting
Consistent binary mask M
Occluded Human
2D pose P
Human Mask M
Inpainting Mask
Figure 4: While generative models provide inconsistent inpainting results, the binary masks that can be extracted from these generated images are much more consistent.
(see Figure 4 middle column). Instead of impainting the occluded parts of the human directly with +, we claim that is more feasible to inpaint binary human masks since small variations in the human silhouette are more acceptable (see Figure rigght column). We first inpaint the RGB image I and then rely on an off-the-shelf segmentation model 23 to obtain the inpainted binary human masks (M), which is used to assist the training of the rendering model in subsequent stages.
4.2 Optimization Stage: Enforcing Human Completeness with SDS Regularization
After obtaining the inpainted masks (M) that outline reasonable human silhouette, we build Gaussian splatting model similar to the one described section 3.2 for human rendering. The 3D Gaussians 11 are initiated as the SMPL mesh vertices, which are able to be deformed adapt to different poses through SMPL-based LBS (Equation 3.1 With the help of (M), the training of II consists of multiple photometric loss terms Lphoto:
A (M--I M.I') + Am L2 (M, A) Assim M-I. AIPs M-I, M-I),, (3)
where L1 is the L-1 loss, L2 is the L-2 loss, SSIM(-) is the SSIM function (56), LPIPS is the VGG- based perceptual loss (77]. I' is the rendered image from 11, and A is the rendered human occupancy map. Each of the loss terms is scaled by weight hyperparameter A.
Even with the supervision (M), geometry inconsistency still exists. Although inconsistent human masks affect the training of IT much less than inconsistent images, human completeness cannot be guaranteed without further steps.
Using diffusion priors to enforce human completeness. We build off of the insights from 59 70 and apply Score Distillation Sampling (SDS) 45 to improve the quality of human renderings and reduce artifacts. Instead of applying SDS on RGB images I', which causes appearance inconsistency. we apply it directly to the rendered human occupancy maps A so that diffusion scores are propagated to encourage complete A:
ØA
SDS
Er,e w(() (co(A:t,P) c)
where is scheduled time stamp, w(() ) a weighting function, c(-) the UNet noise estimator in + and is the injected Gaussian noise.
Using diffusion priors to regularize canonical pose. In-the-wild videos often involve very sparse observations of the human, with only incomplete regions of the human visible in each frame. To propose to render the human in the canonical Da-pose P with
further enforce completeness, the human oriented at random angle E {k: E Z}. Applying SDS on the canonical renderings serves regularization and is randomly activated during training. Overall, at each training step in the Optimization Stage, the 3D Gaussians II are optimized towards:
Vn (photto + Apose SP' + (1 -p)) Aean L
(5)
where / a random variable that has 75% chance to be 1 and otherwise. The Optimization stage results in complete and coherent geometry regardless of the viewing angle.

---

Table 1: Quantitative comparison the CJp- and OcMotion datasets. LPIPS values scaled by x1000. We color cells that have the best and second best metric values.
ZJU-MoCap
44
OcMotion [15]
Methods
PSNR
SSIM
LPIPS
PSNR*1
SSIM
LPIPS J
HumanNeRF
57
20.67
0,9509
3DGS--vaaar
46
17.29f
0,9410
63.25t
9.788t
0.7203t
188. 11
GauHuman
14
21.55
0.9430
55.88
15.09
0.8525
107.1
OccNeRF [64]
22.40
0.9562
43.01
15.71
0.8523
82.90
OccGaussian
167
23.29
0.9482
41.931
Wild2Avatar
14.098
0.84848
93.31
OceGauHuman
22.71
0.9492
54.60
18.85
0,8863
86.53
OccFusion
23.96
0.9548
32.34
18.28
0.8875
82,42
Metries calculated om visible pixels only.
Model trained iterations with training time.
Results taken from OceGaussian [67] using x5 training frames.
Model trained under the default selting 663 using x2 training frames.
4.3 Refinement Stage: Refining Human Appearance via In-context Inpainting
As shown in Figure Exp. and D, applying diffusion priors rendered human occupancy maps is not able to recover the missing appearances of the human. This motivates the need for subsequent stage that keeps refining for better appearance.
The refinement of the appearance of 3D objects not a new topic [53 31 (70. However, no existing generative models capable of handling the consistency of appearance of human across different frames and poses. We attribute this difficulty to the denoising process used in generative priors random noise is injected to rendering each SDS step which leads to uncertain results. This is infeasible for reconstruction tasks, which require frame-consistent representations that agree with all observations.
Our approach focuses on generating inpainted images of the occluded human offline to use as references. We first identify the occluded regions to be inpainted R by using the rendered human occupancy masks A from the Optimization Stage and pre-computed human visibility masks M: R R M) A. In order to encourage the generated regions to be more consistent with the partial observations, we propose in-context references inspired by in-context learning in language models I Although renderings from the Optimization Stage lack sharp and high-fides high-fideitty details, they resemble complete human geometries and possess good enough features that can be used as a coarse reference to guide & to inpaint similar contents at occluded body regions. To achieve this, we stack î and together as single image input to & with an additional prompt phrase "the same person standing in two different rooms".
We use the inpainted RGB images [ī} along with other priors to finetune Il via photometric losses. Since diffusion models still tend to be somewhat inconsistent, we smooth training by putting more weight perceptual loss terms and use L1 loss for the pixel-wise loss terms for its high robustness to variance:
Vn Argb (M I,- I') Amask L2 A) + Agen LI R I') + LPPPS (6)
We train our entire pipeline for only 10 minutes on single TITAN RTX GPU. More implementation details are provided in supplementary materials.
Experiments
In this section, we conduct quantitative and qualitative evaluation our approach against state-of-the- art methods. Then, we conduct ablation studies of our entire pipeline, demonstrating that each stage is necessary for optimal performance. More experiments and results can be found in supplementary materials.

---

Train
ON
OGH
Ours
Ref.
Train
ON
OGH
Ours
Ref.
View
OcMotion Qualitative comparisons on simulated occlusions in the ZJU-MoCap dataset (left
ZJU-MoCap
View
View
View
Figure 5:
column) and real-world occlusions in the OcMotion dataset [16]. ON denotes OccNeRF 64 and OGH denotes OccGauHuman.
5.1 Datasets and Evaluation
Z.UUMoapp ZJU-MoCap 444 is dataset consisting of 6 dynamic humans captured with syn- chronized multi-camera system. Since the humans are in a lab environment free of occlusions, we follow OccNeRF's (64 protocol to simulate occlusion of the human, masking out the center 50% of the human pixels for the first 80 % frames. To challenge OccFusion on videos with even sparser frames, we use only 100 framee from the first camera with sampling rate of train the models and use the other 22 cameras for evaluation.
OcMotion. OcMotion [15] comprises of videos of humans interacting with real objects in indoor environments. Experiments are conducted on the same 6 sequences adopted by Wild2Avatar [63] which are selected to provide a diverse coverage of real-world occlusions. We form sparser sub- sequences by sampling only 50 frames from each sequence to train the models.

---

Table 2: Ablation results the ZJU-MoCap
44
dataset. LPIPS values scaled by x1000.
Exp.
Methods
PSNRT
SSIMI
LPIPS
Train time
GauHuman
14
21.55
0.9430
55.88
10 mins
A
OcmGauHuman
22.54
0.9457
54.88
2 mins
B
Init Stage generated masks {M}
23.52
0.9516
52.35
5 mins
C
Posed space SDS
23.90
0.9510
55.47
7 mins
D
Canonical space SDS (Optim Stage)
23.91
0.9514
55.35
mins
E
Refinement Stage
23.96
0.9548
32.34
10 mins
Evaluation. We compare OccFusion OccNeRF [64]. OccGaussian [67], and Wild2Avatar the state-of-the-ar in occluded human rendering. We also compare our results to GauHuman
63
HemanNeRR [57], and 3DGS-Avatar [46), popular human rendering methods not designed for occlusion. For fairness of comparison, all methods use the same set of segmentation masks and pose priors. We train GauHuman and OceGauHuman for 10 minutes each. We evaluate the methods both quantitatively and qualitatively. For our quantitative evaluations, we calculate the Peak Signal-to-Noise Ratio (PSNR), Structural SIMilarity (SSIM), and Learned Perceptual Image Patch Similarity (LPIPS) metrics against the ground truth images. Since no ground truth is provided for OcMotion, we calculate the metries visible pixels only. For qualitative evaluations, we render the human from novel views and assess the quality of the renderings.
14
5.2 Results Simulated and Real-world Occlusions
We provide quantitative metrics averaged over all the sequences in Table 1 Overall, methods designed for occluded human rendering tend outperform their traditional counterparts. Among those methods, OccFusion consistently performs to par better than the state-of-the-ard on both datasets while significantly beating all the baselines on LPIPS.
Qualitative results on novel view synthesis can be found Figure 5 OccNeRF 664 has trouble generating unseen regions and renders significant discoloration and floaters when faced with occlusion. On the other hand, OceGauHuman 's renderings are blurry and occasionally incomplete. We observe that OccFusion is the only method to consistently render sharp and high-quality renderings free of occlusions.
5.3 Additional Studies
Ablation studies, We study the effect of each of our proposed components by adding them one by one and report average metrics on ZJU-MoCap in Table 2 Each stage plays part towards optimal performance. Qualitative results on our ablations are included in Figure We can see that the Initialization Stage helps enforce completeness for the initially incomplete human. The SDS regularization provided in the Optimization Stage helps remove floaters and artifacts in the posed and canonical space, further improving the shape of the human and enforcing completeness of the body. Finally, the Refinement Stage helps make the renderings more detailed in less observed regions, improving the rendering quality and greatly reducing the LPIPS.
Input
Exp. A
Exp.
Exp.
Exp. E
Ref.
Exp.
Exp. C
Exp.
Figure 6: Qualitative ablation studies.
Please see Table for corresponding experiments. Major
differences are highlighted by red arrows.

---

Does the proposed OceGauHuman perform better than GauHuman [4] in rendering occluded humans? In section 3.3 present simple upgrade for the state-of-the-art 3DGS based human rendering model GauHuman [14] to help it better handle occlusions. Our improvements are straight- forward but effective. We show quantitative results in Figure (Left) and Table As shown in Figure / our improved OccGauHuman reconstructs more complete human body than the vanilla GauHuman.
Input
GH
OGH
Ours
Ref.
Input
GH
OGH
Ours
Ref.
ZJU-MoCap
OcMotion
Figure 7: Qualitative comparisons simulated occlusions in the ZJU-MoCap dataset column) and real-world occlusions in the OcMotion dataset [16] (right column). GH denotes GauHuman [[4] and OGH denotes OceGauHuman.
[44 (left
Discussions and Conclusion
Limitations. Recovering occluded dynamic humans is challenging, As mentioned in section 4.3 reconstructing a 3D human requires adhering to multiple consistencies. However, even with the state-of-the-art generative models, it is still impossible to perfectly maintain those consistencies for 4D content (3D + motion) generation. Although our proposed methods are specifically designed to eliminate potential variances when using generative priors, we can still observe some generations are less coherent (e.g. Figure and Figure 8). which may hurt the training of the rendering model on all stages. Moreover, we found that conditioning generative models with 2D poses is weak the pose of the generated human does not always align with the condition pose, which may introduce even more uncertainty for training. In future work, we hope to train our own consistency- diffusion model specifically finetuned on human data.
Societal Impacts. Being able reconstruct human from occluded monocular video can have great societal impact. For example, having a high- fidelity 3D reconstruction of a human can help telemedicine practitioners become more immersed in the 3D space. While research could lead to privacy concerns if humans are reconstructed without their consent, we believe that the benefits can be harnessed responsibly with appropriate safeguards.
Conclusion. In this work, we propose OccFusion, one of the first works that utilize 3D Gaussian splatting for occluded human rendering. Our approach consists three stages: the Initialization, Optimization, and Refinement stages. By combining the efficiency and representative ability of 3D Gaussian splatting with the generation capabilities of diffusion priors, our method achieves tfate-of-th--rr in occluded human rendering quality as measured by the PSNR, SSIM, and LPIPS metrics while only taking around 10 minutes train. We hope our work inspires further exploration into the capabilities of diffusion priors to aid in human reconstruction.
10

---

7 Acknowledgment
This work was partially funded by the NIH Grant RO1AG089169 and P41EB027060, Panasonic Holdings Corporation, the Gordon and Betty Moore Foundation, the Jaswa Innovator Award, Stanford HAI, Stanford HAI graduate fellowship, and Stanford Wu Tsai Human Performance Alliance.
References
[1] Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subhiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Language models are few-shot leamnrrs. Advances in neural information processing systems, 33:1877-1901, 2020.
[2] Mingfei Chen, Jianfeng Zhang, Xiangyu Xu, Lijuan Liu, Yujun Cai, Jiashi Feng, and Shuicheng Yan. Geometry-guidee progressive nerf for generalizable and efficient neural human rendering. In European Conference on Computer Vision, pages 222-239. Springer, 2022.
[3] Xingyu Chen, Qi Zhang, Xiaoyu Li, Yue Chen, Ying Feng. Xuan Wang, neural radiance fields in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12943-12952, 2022.
Wang. Hallucinated
[4] Alvaro Collet, Ming Chuang, Pat Sweeney, Don Gillett, Dennis Evseev, David Calabrese, Hugues Hoppe, Adam Kirk, and Steve Sullivan. High-quality streamable free-viewpoint vidco. ACM Transactions on Graphics (ToG), 34(4):1-13, 2015.
[5] Mingsong Dou, Sameh Khamis, Yury Degtyarev, Philip Davidson, Sean Ryan Fanello, Adarsh Kowdle, Sergio Orts Escolano, Christoph Rhemann, David Kim, Jonathan Taylor. et al. Fusion4d: Real-time performance capture of challenging scenes. ACM Transactions on Graphics (ToG). 35(4):1-13, 2016.
[6] Chen Geng, Sida Peng, Zhen Xu, Hujun Bao, and Xiaowei Zhou. Learning neural volumetric representa- tions dynamic humans in minutes. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 8759-8770, 2023.
[7] Rohit Girdhar, Mannat Singh, Andrew Brown, Quentin Duval, Samaneh Azadi, Sai Saketh Rambhatla, Akbar Shah, Xi Yin, Devi Parikh, and Ishan Misra. Emu video: Factorizing text-to-video generation by explicit image conditioning. arXiv preprint arXiv:211..1000, 2023.
[8] Shubham Goel. Georgios Pavlakos, Jathushan Rajasegaran, Angjoo Kanazawa, and Jitendra Malik. Hu- mans in 4d: Reconstructing and tracking humans with transformers. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 14783-14799 2023.
[9] Chen Guo, Tianjian Jiang, Xu Chen, Jie Song, and Otmar Hilliges. Vid2avatar: 3d avatar reconstruction from videos the wild via seff-supervised scene decomposition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12858-12868, 2023.
[10] Kaiwen Guo, Peter Lincoln, Philip Davidson, Jay Busch, Xueming Yu, Matt Whalen, Geoff Harvey, Sergio Orts-Fseolano, Rohit Pandey, Jason Dourgarian, al. The relightahles: Volumetric performance capture of humans with realistic relighting, ACM Transactions on Graphics (ToG), 38(6):1-19, 2019.
[11] Agrim Gupta. Lijun Yu, Kihyuk Sohn, Xiuye Gu, Meera Hahn, Li Fei-Fei, Irfan Essa, Lu Jiang, and José Lezama. Photorcalistic video generation with diffusion models. arXiv preprint arXiv:
2023.
[12] Liangxiao Hu, Hongwen Zhang, Yuxiang Zhang, Boyao Zhou, Boning Liu, Shengping Zhang, and Liqiang Nie. Gaussianavatar: Towards realistic human avatar modeling from single video via animatable 3d gaussians. arXiv preprint 2rx 2023.
[13] Shoukang Hu, Fangzhou Hong, Liang Pan, Haiyi Mei, Lei Yang, and Ziwei Liu. Sherf: Generalizable human nerf from a single image. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9352-9364, 2023.
[14] Shoukang Hu and Ziwei Liu, Gauhuman: Articulated gaussian splatting from monocular human videos. arXiv preprint arXiv:2312.02973. 2023.
[15] Buzhen Huang, Yuan Shu, Jingyi Ju. and Yangang Wang. Occluded human body capture with self- supervised spatial- temporal motion prior. arXiv preprint arX1v:2207.05375, 2022.
11

---

[16] Buzhen Huang. Tianshu Zhang, and Yangang Wang. Object-occluded human shape and pose estimation with probabilistic latent consistency. IEEE Transactions Pattern Analysis and Machine Intelligence. 2022.
[17] Boyi Jiang. Yang Hong, Hujun Bao, and Juyong Zhang. Selfrecon: Self reconstruction, your digital avatar from monocular video. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 5605-5615, 2022.
[18] Tianjian Jiang, Xu Chen, Jie Song, and Oumar Hilliges. Instantavatar: Learning avatars from monocular video 60 seconds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16922-16932, 2023.
[19] Wei Jiang, Kwang Moo Yi, Golnoosh Samei, Oncel Tuzel, and Anurag Ranjan. Neuman: Neural human radiance field from a single video. In European Conference on Computer Vision, pages 402-418. Springer, 2022.
[20] HyunJun Jung, Nikolas Brasch, Jifei Song, Eduardo Perez-Pellitero, Yiren Zhou, Zhihao Li, Nassir Navab, and Benjamin Busam. Deformable 3d gaussian splatting for animatable human avatars. arXiv preprint S.ki::2312.15059 2023.
[21] Animesh Karnewar, Niloy Mitra, Andrea Vedaldi, and David Novotny. Holofusion: Towards photo- realistic 3d generative modeling. In Proceedings of the IEFF/CVF International Conference Computer Vision, pages 22976-22985, 2023.
[22] Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Metzger, Rodrigo Caye Daudt, and Konrad Schindler. Repurposing diffusion-based image generators for monocular depth estimation. arXiv preprint arXiv:2312.02145.222 2023.
[23] Lei Ke, Mingqiao Ye, Martin Danelljan, Yu- Yu-Wing Tai, Chi- Chi-Keeng Tang, Fisher Yu, et al. Segment anything in high quality. Advances Neural Information Processing Systems, 36, 2024.
[24] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4):1-14, 2023.
[25] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitchead, Alexander C Berg, Wan-Yen Lo, al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4015-4026, 2023.
[26] Muhammed Kocabas, Jen-Hao Rick Chang, James Gabriel, Oncel Tuzel, and Anurag Ranjan. Hugs: Human gaussian splats. arXiv preprint arXiv:2311.1799 2023.
[27] Youngjoong Kwon, Dahun Kim, Duygu Ceylan, and Henry Fuchs. Neural human performer: Learning generalizable radiance fields for human performance rendering. Advances Neural Information Processing Systems, 34:24741 -247522, 2021.
[28] Mengtian Li, Shengxiang Yao, Zhifeng Xie, Keyu Chen, Yu- fu-Gang Jiang. Gaussianbody: Clothed human reconstruction via gaussian splatting. arXiv preprint arXiv:2401.097. 2024.
[29] Mingwei Li, Jiachen Tao, Zongxin Yang, and Yi Yang. Human101: Training 100+ fps human gaussians in 100s from 1 view. arXiv preprint arXiv:2312. 15258, 2023.
[30] Chen-Hsuan Lin, Jun Gao, Luming Tang. Towaki Takikawa, Xiaohui Zeng. Xun Huang, Karsten Kreis, Sanja Fidler, Ming- Liu, and Tsung- Yi Lin. Magic3d: High-resolution text-to-3d content creation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patiern Recognition, pages 300-309, 2023.
[31] Yuanze Lin, Ronald Clark, and Philip Torr Dreampolisher: Towards high-quality text to-3d generation via geometric diffusion. arXiv preprint arXiv:2403. 17237, 2024.
[32] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero- 1-to-3: Zero-shot one image to 3d object. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9298-9309, 2023.
[33] Yang Liu, Xiang Huang. Minghan Qin, Qinwei Lin, and Haoqian Wang. Animatable 3d gaussian: Fast and high-quality reconstruction of multiple human avatars. arXiv preprint 2rXiv:2311.16449 2023.
[34] Yuan Liu, Cheng Lin, Zijiao Zeng, Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang. Synedreamer: Generating multiview-consiser images from a single-view image. arXiv preprint arXiv:2309.0344 0343339990333..... 2023,
12

---

[35] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael Black. Smpl: A skinned multi-person linear model. In Seminal Graphics Papers: Pushing the Boundaries, Volume 2, pages 851-866. 2023.
[36] Ricardo
Noha Radwan, Mehdi SM Sajjadi, Jonathan T Barron, Alexey Dosovitskiy, and Daniel Duckworth. Nerf in the wild: Neural radiance fields for unconstrained photo collections. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7210-7219, 2021.
[37] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis, Communications of the ACM, 65(1)999-106, 2021.
[38] Arthur Moreau, Jifei Song, Helisa Dhamo, Richard Shaw, Yiren Zhou, and Eduardo Pérez-Pellitero Human gaussian splatting: Real-time rendering of animatable avatars, arXiv preprint arXiv:2311.17111,, 2023.
[39] Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2i- adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In Proceedings of the AAAl Conference on Artificial Intelligence, volume 38, pages 4296-4304, 2024.
[40] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM transactions on graphics (TOG), 41(4):1-13 15, 2022.
[41] Xiao Pan, Zongxin Yang, Jianxin Ma, Chang Zhou, and Yi Yang. Transhuman: A transformer-hasee human representation for generalizable neural human rendering. In Proceedings of the IEEE/CVF International conference compuder vision, pages
2023.
[42] Haokai Pang, Heming Zhu, Adam Kortylewski, Christian Theobalt, and Marc Habermann. Ash: Animatable gaussian splats for efficient and photoreal human rendering. arXiv preprint arXiv:2312.05941, 2023.
[43] Bo Peng, Jun Hu, Jingtao Zhou, Xuan Gao, and Juyong Zhang, Intrinsiengg: Intrinsic coordinate based hash encoding for human nerf. IEEE Transactions Visualization and Computer Graphics, 2023.
[44] Sida Peng. Yuanqing Zhang, Yinghao Xu, Qianqian Wang, Qing Shuai, Hujun Bao, and Xiaowei Zhou. Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dy- namic humans. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9064-9063. 2021.
[45] Ben Poole, Ajay Jain, Jonathan Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209. 149888 2022.
[46] Zhiyin Qian, Shaofei Wang, Marko Mihajlovic, Andreas Geiger, and Siyu Tang. 3dgs-avatar: Animatable avatars via deformable 3d gaussian splatting. arXiv preprint arXiv: 2023.
[47] Weining Ren, Zihan Zhu, Boyang Sun, Jiaqi Chen, Mare Pollefeys, and Songyou Peng. Nerf on on-the-go: Exploiting uncertainty for distractor- free nerts in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8931-8940, 2024.
[48] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10699 2022.
[49] Ruizhi Shao, Zerong Zheng. Hongwen Zhang, Jingxiang Sun, and Yebin Liu. Diffustereo: High quality human reconstruction via diffusion- bastud stereo using sparse cameras. In European Conference on Computer Vision, pages 702-720. Springer, 2022.
[50] Zhuo Su, Lan Xu, Zerong Zheng, Tao Yu, Yehin Liu, Lu Fang. Robustfusion: Human volumetric capture with data-driven visual cues using a rgbd camera. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IV 16, pages 246-264. Springer, 2020.
[51] Guoxing Sun, Xin Chen, Yizhang Chen, Angi Pang, Pei Lin, Yuheng Jiang, Lan Xu, Jingyi Yu, and Jingya Wang. Neural free-viewpoint performance rendering under complex human-object interactions. In Proceedings of the 29th ACM International Conference on Multimedia, MM '21, page 4651 4651áA 14A66660, New York, NY, USA, 2021. Association for Computing Machinery.
[52] Wenzhang Sun, Yunlong Che, Han Huang, and Yandong Guo. Neural reconstruction of relightable human model from monocular video. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 397-407. 407, 2023.
13

---

[53] Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. arXiv preprint arXiv:2309.1663. 2023.
[54] Guangeong Wang, Zhaoxi Chen, Chen Change Lay, and Ziwei Liu, Sparsenerf: Distilling depth ranking for few-shot novel view synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9065-9076, 2023.
[55] Haochen Wang. Xiaodan Du, Jiahao Li, Raymond A Yeh, and Greg Shakhnarovich. Score jacobian chaining: Lifting pretrained 2d diffuusin models for 3d generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26619-16229 2023.
[56] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4);600-61) 2004.
[57] Chung-Y Yi Weng, Brian Curless, Pratul P Srinivasan, Jonathan Barron, and Ira
Shlizeerann- Humannerf: Free-viewpoint rendering of moving people from monocular video. In Proceedings of the JEEE/CVF conference on compiter vision and pattern Recognition, pages 16210-16220, 2022.
[58] Zhenzhen Weng, Jingyuan Liu, Hao Tan, Zhan Xu, Yang Zhou, Serena Yeung-Levy, and Jimei Yang. Single-view 3d human digitalization with large reconstruction models. arXiv preprint MkKiv:2401.12173. 2024.
[59] Zhenzhen Weng, Zeyu Wang, and Serena Yeung. Zeroavatar: Zero-shot 3d avatar generation from a single image. arXiv preprini arXiv:2305. 16411, 2023.
[60] Jay Zhangjic Wu, Yixiao Ge. Xintao Wang. Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-vided One-shot tuning of image diffusion models for text-to-video generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages (633-7633, 2023.
[61] Rundi Wu, Ben Mildenhall, Philipp Henzler. Keunhong Park, Ruiqi Gao, Daniel Watson, Pratul Srini- vasan, Dor Verbin, Jonathan T Barron, Ben Poole, et al. Reconfusion: 3d reconstruction with diffusion priors. arXiv preprint ar arXiv:2312.02990 2023.
[62] Jamie Wynn and Daniyar Turmukhambetov. Diffusionerf: Regularizing neural radiance fields with denoising diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4180-44 0000-119,,200 2023.
[63] Tiange Xiang, Adam Sun, Scott Delp, Kazuki Kozuka, Li Fei-Fei, and Ehsan Adeli. Wild2avatar: Rendering humans behind occlusions. arxiv preprint arXiv:2401.004 004311 2023.
[64] Tiange Xiang, Adam Sun, Jiajun Wu, Ehsan Adeli, and Li Fei-Fei. Rendering humans from object-o monocular videos. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3239-3250 2023.
[65] Jiale Xu, Weihao Cheng. Yiming Gao, Xintao Wang, Shenghua Gao, and Ying Shan. Instantmesh: Efficient mesh generation from single image with sparse- view large reconstruction models. arXiv preprint arXiv:2404.07199 2024.
[66] Xiaofeng Yang, Yiwen Chen, Cheng Chen, Chi Zhang. Yi Xu, Xulei Yang. Fayao Liu, and Guosheng Lin. Learn to optimize denoising scores for generation: A unified and improved diffusion prior on nerf and 3d gaussian splatting. arXiv preprint 23 2023.
[67] Jingrui Ye, Zongkai Zhang, Yujiao Jiang. Qingmin Liao, Wenming Yang, and Zongqing Lu. Ocegaussian: gaussian splatting for occluded human rendering. arXiv preprint arXiv:2404.08445 2024.
[68] Keyang Ye, Tianjia Shao, and Kun Zhou, Animatable 3d gaussians for high-fidelity synthesis of human motions. arXiv preprint 231 1. 13404, 2023.
[69] Vickie Ye, Georgios pavlakos, Jitendra Malik, and Angjoo Kanazawa. Decoupling human and camera motion from videos in the wild. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 21222-21233 2023.
[70] Taoran Yi. Jiemin Fang. Junjie Wang, Guanjun Wu, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Qi Tian, and Xinggang Wang. Gaussiandreamer: Fast generation from text to 3d gaussians by bridging 2d and 3d diffusion models, 2024.
[71] Tao Yu, Zerong Zheng, Kaiwen Guo, Pengpeng Liu, Qionghai Dai, and Yebin Liu. Function4d: Real-time human volumetric capture from very sparse consumer rgbd sensors. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 5756, 2021.
14

---

[72] Zhengming Yu, Wei Cheng. Xian Liu, Wayne Wu, and Kwan- Lin. Monohuman: Animatable human neural field from monocular video. In Proceedings the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 16943-16953, 2023.
[73] Zhongrui Yu, Haoran Wang. Jinze Yang, Hanzhang Wang, Zeke Xie, Yunfeng Cai, Jiale Cao, Zhong Ji and Mingming Sun. Sgd: Street view synthesis with gaussian splatting and diffusion prior. arXiv preprint arXiv:2403.2079,22222 2024.
[74] Yuan, Xueting Yangyi Huang, Shalimi De Mello, Koki Nagano, Jan Kautz, Umar lqbal. Gavatar: Animatable gaussian avatars with implicit mesh learning. arXiv preprint arXiv:2312.1142 2023.
[75] Dongbin Zhang. Chuming Wang, Weitao Wang, Peihao Li, Minghan Qin, Haoqian Wang. Gaussian in the wild: 3d gaussian splatting for unconstrained image collections. arXiv preprini 15X14:2403.15004
2024.
[76] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 3836-3847, 2023.
[77] Richard Zhang, Phillip Isola, Alexci A Efros, Eli Shechtman, and Oliver Wang. The unrcasonable effectiveness of deep features perceptual metrie. In CVPR, 2018.
[78] Tingyang Zhang, Qingzhe Gao. Weiyu Li, Libin Liu, and Baoquan Chen. Bags: Building animatable gaussian splatting from monocular video with diffusion priors. arXiv preprint arXiv:2403.1 /427, 2024.
[79] Zhizhuo Zhou and Shubham Tulsiani. Sparsefusion: Distilling view niw-cconditioned diffusion for 3d recon- struction. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 12588-12597, 2023.
15

---

A Table of Symbols
For notation simplicity, we adopted alphabetic symbols this paper to represent essential components in our framework. For better symbol-name correspondences, here we justify the implications of all symbols used in the paper in Table
Table 3: Table of symbols.
Symbols
Explanations
Preliminaries
X
3D points the canonical human space 3D points in the posed human space skinning weights used in LBS transformation matrix used in LBS translation vector used in LBS 3D locations of human joints pose parameters used in SMPL 33
W
G
J
shape parameters used in SMPL 35
center of a 3D Gaussian opacity of 3D) Gaussian
mean value of 3D Gaussain
Σ
covariance matrix of 3D Gaussain
OceFusion
optimizable human 3D Gaussians a pretrained generative model 48]. used as prior precomputed binary human mask, used as prior precomputed human pose, used as prior
M
P
P
the canonical articulation of P input image with occluded human Init Stage generated complete human mask SDS gradients, used as a guidance in the Optim Stage
M
A
I
Optim Stage rendered human RGB image IT rendered human occupancy map in all stages Refine Stage rendered human RGB image inpainting mask computed by (1 -1-10 (1-M)- A random variable [0, controls Optim Stage SDS
R
Implementation Details
OccFusion requires several priors. We run SAM 25 to get all the human masks {M). While we follow previous work [64 [3] and use the ground truth poses provided by ZJU-MoCap and OcMotion. pose priors P can be obtained via occlusion-robust SMPL prediction/opimizatioo methods such as HMR 2.0 [8] and SLAHMR [69 for in-the-wild videos. Improving the quality of priors is not the focus of this work. We use the pre-trained Stable Diffusion 1.5 model (48 with ControlNet (76 plugins for SDS in all the stages.
In the Initialization Stage, instead of inpainting incomplete human masks directly, we run the pretrained diffusion model inpaint RGB images with 10 inference steps and 1.0 ControlNet conditioning scale. We use the positive prompt- "clean background, high contrast to the background, person only, plain clothes, simple clothes, natural body, natural limbs, no texts, overlay" and the negative prompt- ""multiple objects. occlusions, complex pattern, fancy clothes, longbody, lowres, bad anatomy, bad hands, bad feet, missing fingers, cropped, worst quality, low quality, blurry". After inpainting the RGB images, we then run SAM-HQ 23 with P as the prompts get (M).
the Optimization Stage, we train the human Gaussian IT from scratch by following the objective Equation 5) We set Argb = led Amask = 2e Ausim = 3e3 and Alpips 1 At each training step, we random switch the SDS regularization on either posed human space or the canonical Da-pose space with probability of 75% and 25%. When applying SDS regularization on the canonical human
16

---

space, we randomly rotate the human horizontally with uniformly sampled degree in {kī A }}. We set the SDS loss weights as Apose = 2e and Acan = 2c In this stage, we train IT for 1200 steps.
In the Refinement Stage, we first generate the RGB human inpaintings via the proposed in-context inpainting method. We run the pretrained diffusion model with conditions on M, 10 inference steps, and 0.3 ControlNet conditioning scale. We did not use positive prompts for the inpainting but used the same negative prompts as in the Optimization Stage. During training, we set the loss weights as Argb and Amask 0.1. Agen 0.1, and Aipips 0.2. In this stage, we finetune II for another 1800 steps with Gaussian densification and pruning enabled for the first 1000 steps.
C Additional Studies
Effectiveness of in-context inpainting. We provide comparisons the human in the Refinement Stage with and without in-context inpainting and provide qualitative comparisons in Figure X While renderings from the Optimization stage are less detailed in occluded areas, our proposed in-context inpainting is able to generate the missing content and greatly increase the rendering quality in these areas.
Decueed Hamtin
Ontin Soogs Rendbringg
wit In-corot iep
w/ moneit Irponning
Unoccatd References
Figure 8: Comparison of the inpainted human the Refinement Stage with and without using the proposed in-context inpainting technique. Major differences are highlighted with red arrows.
ENVTERREST
Optim. Stage
Refine. Stage
Optim. Stage Refine. Stage
Train with RGB SDS
Train with mask SDS
Figure 9: Applying SDS RGB images vs. on human occupancy maps. As mentioned in Sec. 4.1 and Fig. 4 of the main paper, generated RGB appearances are much more inconsistent than generated silhouettes. As a result, applying SDS on RGB leads defective rendering results.
17

---

Optim. Stage
Refine. Stage
Optim. Stage
Refine. Stage
Train with complete masks Figure 10: Training with complete unoccluded masks vs. with inpainted masks in the Optim. stage. Although inpainted masks are slightly more inconsistent compared the complete masks, our training pipeline converges to the same level of rendering quality.
Train with in-painted _aasss
Input image condition
Novel view synthesis
Figure 11: Novel view synthesis results from InstantMesh 65 conditioned on the least occluded
frame. Discrepancies are circled in red.
Applying SDS on RGB vs on Human Occupancy Maps. We include additional experiments comparing the rendering results of applying SDS on RGB proposed). is clear that applying SDS on RGB leads to defective renderings as well as inferior quantitative results. This experiment validates our claim made in Sec. 4.1 and Fig. 4 in the main paper.
human occupancy maps (as
Robustness of training to inpainted masks. For in-the-wild occluded videos, there ground truth masks for the occluded body regions due to unknown juman//arment deformations. Relying on the state-of-the-au pre-trained priors brought by the Segment Anything model (SAM) 25 and Stable Diffusion 48). the segmented/inpaiitee masks are expected to be reasonable and coherent across frames. To test the robustness of our method to variances the in-painted masks, we add comparison experiments on ZJU-MoCap that supervise using the complete SAM masks obtained from the unoccluded humans with minimum variances. Please see the qualitative results in Figure IO We find that using the inpainted M leads to a good enough rendering quality comparable to using masks derived from the unoccluded images, validating the robustness of our model.
18

---

Can existing generative models recover an occluded human? While there are works for using generative diffusion models to render 3D humans conditioned on single [58] and multiple 49 images, none are able to condition on monocular video of the person.
We use the provided seg- mentation mask to mask the least occluded frame onto a white background and use it as conditioning. Novel view synthesis results are included in Figure [] InstantMesh unable to recover a complete human geometry and fails to generate reasonable appearance from the single image.
Since [58] has released code, include results from InstantMesh
65
D
Video Studies
For a more comprehensive presentation of the results, we include video renderings on all the training frames for both datasets. For the ZJU-MoCap videos (named with the prefix zju), from left to right, we show the occluded human, OceGauHuman rendering, Optimization Stage rendering, Refinement Stage rendering, and the reference. For the OcMotion videos (named with the prefix ocmotion), without references for real-world occlusions, from left to right, we show the occluded human, OceGauHuman rendering, Optimization Stage rendering, and Refinement Stage rendering.
19

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: Our work proposes novel stage pipeline for rendering occluded humans with Gaussian splatting and diffusion priors that outperforms the state-of-the-art in the field.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Our paper includes a "Limitations" section Section 6 that acknowledges the inconsistency of generative diffusion models, which are central approach.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-tex system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
20

---

Answer: [NA]
Justification:
Our paper does not include any new theorems, formulas, or proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: ZJU-MoCay and OcMotion, the datasets used in this paper. publicly available, and we include all implementation details in supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path reproducing verifying the results. 5. Open access to data and code
21

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We are not including code in our submission. However, we provide implemen- tation details in our supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so âÄlJNoâÄ is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed run reproduce the results. See the NeurlPS code and data submission guidelines (https
/ttie////guidee/eeeuuuubiiiii ssi osoonicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Implementation details are all included in supplementary materials. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars are not reported, since they are not a common practice in the human rendering research community.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
22

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We mention the hardware used to train our method at the end of Section We also break down the amount of time needed to train each stage in Table 2
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics htttps://neuris..... http://////eerpps.ccccc/c/////// http:/pp.....p..c.tctccccccceeee
Answer: [Yes]
Justification: The authors have reviewed the NeurIPS Code of Ethics and strongly believe it conforms to the Code.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We include discussion of societal impacts in
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
23

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: Our method utilizes per-video optimization, no pretrained weights provided and no models or data are released.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: In Subsec
cite GauHuman in our paper and the license used, and provide a link to the code.
3.3
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: New assets are not used or included in our submission.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our work does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Our work does involve crowdsoureing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
25

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "nuwDNE4EOe", "reviewer": "Reviewer_gVDG", "summary": "This work presents a Gaussian-based approach to reconstruct 3D human poses from occluded monocular video. Building upon the success of previous methods like SDS, the paper suggests introducing a pre-trained diffusion prior to complete the occluded areas and further refine the missing appearance. The framework is evaluated on public datasets, including ZJU-MoCap and OcMotion, and shows improved efficiency and appearance quality.", "questions": "- The supplementary videos lack method names, making it difficult to identify the final result of the proposed \"OccFusion\" method. It is unclear why some blur exists in the visible areas, as evident in cases like \"ocmotion_0011_1\" and \"ocmotion_0011_2\".", "limitations": "No negative societal impact.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This paper tackles a relatively new and practical problem of 3D human reconstruction from occluded monocular video.\n- The introduction section provides a clear motivation and background for the proposed method.\n- The authors conduct comprehensive experiments to validate the efficiency of each component of the framework.", "weaknesses": "- The ablation study could be more extensive. The authors mention that \"applying SDS on RGB images causes appearance inconsistency\" in line 192. It would be helpful if the authors could provide some visualizations to support this claim.\n- If the optimization step can generate consistent appearances, the refinement stage could also introduce inconsistencies in the occluded areas, as there is no cross-frame constraint during the inpainting process.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for your time and the helpful comments! We address your concerns below.\n\n> Proof that applying SDS on RGB images causes appearance inconsistency\n\nIn Fig. 3 of the rebuttal PDF, we include additional experiments comparing the rendering results of applying SDS on RGB vs. on human occupancy maps (as proposed). It is clear that applying SDS on RGB leads to defective renderings in the Optimization stage, which ultimately cause inferior results in the Refinement stage. We attribute these defective renderings to the variances introduced from diffusion models, making it difficult for them to reconstruct a consistent human appearance across different frames and training steps. However, applying SDS on human occupancy maps does not suffer from these variances too much and allows for the enforcing of human completeness, as proposed.\n\n> Refinement stage could introduce cross-frame inconsistency\n\nIt is true that diffusion-based inpainting models suffer from inconsistencies. However, $\\{\\mathbf{\\hat{I}}\\}$, the renderings from the optimization stage, are consistent since they are recovered directly from the input video. As we use these consistent images as context for our inpainting, the resulting inpainted reference images are not dramatically inconsistent. It is also worth noting that any small inconsistencies present in the inpainted references are smoothed out by more weight on perceptual loss terms and the usage of L1 loss during optimization of the Refinement stage (lines 227-230 of the main paper).\n\n>Supplementary videos lack method names\n\nSorry for the inconvenience. We included the naming details in section “D” of the supplementary material, but we can see how it can be confusing. We will be sure to label the videos with method names in the final version.\n\n> Blur present in visible areas\n\nWith only 10 mins of training time, OccFusion surpasses state-of-the-art occluded human rendering methods by a significant margin qualitatively and quantitatively. In addition, with longer training time, OccFusion is able to render in higher quality with much less blur. We add experiments to validate this statement – please see qualitative and quantitative results in Fig. 1 in the rebuttal PDF. We believe that our proposed 10 minute version of OccFusion is able to achieve the best balance of rendering quality and efficiency.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XFChRi0N0h", "reviewer": "Reviewer_9Cmr", "summary": "OccFusion proposes an approach to model human bodies that fail under occlusion in monocular videos. The authors utilize 3D Gaussian splatting for efficient rendering and leverage pretrained off-the-shelf image diffusion models as 2D priors. Their approach involves a three-stage training process, sequentially refining estimates to achieve precise 3D human representations.  They demonstrate superior performance compared to previous methods, enhancing one of the baselines to better handle occlusion scenarios.", "questions": "I would like authors to answer to the issues I raise in Weaknesses. If I find them constructive, I would raise the rating.", "limitations": "-", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. The paper is well-written and provides valuable engineering insights, particularly through the extensive use of various off-the-shelf models. The authors demonstrate proficient application of these methods to their tasks. \n\n2. As for another strength, they enhance the GauHuman baseline to handle occlusion correctly, a significant contribution worth highlighting. \n\n3. The ablation study is informative and visual, effectively supporting the narrative. \n\n4. The qualitative study shows the method's superiority, with well-justified reasons in the text.", "weaknesses": "1. Some important details are either omitted or mentioned too late in the text. Below are some examples:\n\n1.1. While the extensive use of off-the-shelf models is well justified, it is only superficially discussed in the corresponding technical sections. For instance, the introduction does not mention the use of image generative models at the initialization stage to improve the masks. This is a core aspect of the work, and mentioning models like DreamFusion, Stable Diffusion, and ControlNet should be done when discussing the three stages, preferably in the introduction.\n\n1.2. Figure 2 (method overview) is cumbersome and not very instructive. It does not clarify the three-stage training, and its understanding relies heavily on the detailed ablation study. I recommend reorganizing Figure 2 to include all off-the-shelf models used in the training pipeline. Even a plain list of inputs and outputs with the side models used at each stage would be more descriptive than the current Figure 2 and the introductory motivation.\n\n\n2. Obscure training details\n\n2.1 Training time: authors insist throughout the text that the training time of their method is 10 minutes. Assuming the monocular video is the only input, the result must depend on the length of the video, yet it is never discussed. \n\n2.2 It is unclear why 10 minutes is enough for good performance. Would estimates improve if trained longer?\n\n\n3. Results\n\n3.1 In Fig.6 Exp.A authors demonstrate that OccGauHuman estimate has some bodyparts ripped apart. This should not happen if initial splats are distributed all over SMPL mesh and \"densification and pruning of splats is disabled. (line 137)\". How can authors explain this behavior? \n\n3.2 In Table 2 GauHuman and OccGauHuman are trained for 10 and 2 minutes, correspondingly. This is unclear how estimates improve when OccGauHuman is trained for 10 minutes, as GauHuman. \n3.3 Continuing the issue 3.2, if all experiments with OccGauHuman model are also taken with 2-min limit, this makes the quantitative comparison unfair. maybe it is simply undertrained.\n\n\n4. (minor) OccGauHuman. In Sec.3 authors should say explicitly that this method they propose is not the part of the contribution but the way to build a better baseline and a starting point for their own method.\n\n5. (minor) Throughout the paper authors say \"gaussian\" referring to the splat or blob. It is better to avoid such namedropping, since Gaussian is a very broad term that can imply many different things and is itself an adjective and does not fit in the sentence properly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for your time and the helpful comments! We address your concerns below.\n\n> Discussion of off-the-shelf models\n\nThanks for the suggestion. We will include the following edited text in the Introduction of the final version:\n\n```\nIn this work, we introduce OccFusion, an efficient yet high quality method for rendering occluded humans. To gain improved training and rendering speed, OccFusion represents the human as a set of 3D Gaussians. Like almost all other human rendering methods, OccFusion assumes accurate priors such as human segmentation masks and poses are provided for each frame, which can be obtained with state-of-the-art off-the-shelf estimators such as SAM and HMR 2.0. However, to ensure complete and high-quality renderings under occlusion, OccFusion proposes to utilize generative diffusion priors, more specifically pose-conditioned Stable Diffusion 1.5 with ControlNet plugins, to aid in the reconstruction process.\n```\n\n> Instructiveness of Figure 2\n\nThanks for your suggestions. We will carefully revise Figure 2 to incorporate your comments.\n\n> Effect of length of video on training time\n\nThe training time of the video does indeed depend on the length of the video. As mentioned in section 5.1 of the paper, we train our model on 100 frames for ZJU-MoCap and 50 frames for OcMotion. We also show in Figure 1 that our model outperforms some baselines that require much more (>500) frames and much longer training time. With longer training time and/or more samples from the videos, OccFusion is able to provide even better results (see Fig. 1 in the rebuttal PDF).\n\n> Does training longer improve rendering results?\n\nYes. With only 10 mins of training time, OccFusion already surpasses all counterparts by a significant margin qualitatively and quantitatively. It is also true that with longer training time, OccFusion is able to render higher quality. We add experiments to validate this statement, please see the qualitative and quantitative results in Fig. 1 in the rebuttal PDF. We believe that our proposed 10 minute version of OccFusion is able to achieve the best balance of rendering quality and efficiency.\n\n> OccGauHuman has splats that are “ripped apart”\n\nAlthough the number of 3D Gaussians does not change during training, as densification and pruning of splats are disabled, the mean, scale, and opacity of each 3D Gaussian is optimized and changed during training, resulting in the “ripped apart” appearance when rendered in 2D. This is because 3D Gaussians are crowded in the visible regions. This ripped apart appearance only occurs in occluded areas, demonstrating that OccGauHuman alone is unable to reconstruct unobserved areas due to its lack of generative capabilities.\n\n> Training time of OccGauHuman\n\nThe OccGauHuman model we use to compare to GauHuman and OccFusion in Table 1 is trained for 10 minutes. We will be sure to clarify this in the final version. \n\nIt is also worth noting that GauHuman and OccGauHuman do not benefit significantly from training for longer. We show in Fig. 1 of the main paper that increasing the training time from 2 to 10 minutes causes only a small boost in PSNR on ZJU-MoCap for OccGauHuman and actually causes a degradation in performance for GauHuman. Additional results in Fig. 1 of the rebuttal PDF also validate this.\n\n> OccGauHuman is not a contribution\n\nThanks for the suggestion. We will rephrase Sec 3.3 of our paper to clarify that we do not consider OccGauHuman a contribution but rather a better baseline.\n\n> Gaussians are a broad term\n\nThanks for the suggestion. We will follow current Gaussian splatting literature [1,2] and replace all mentions of “Gaussians” in our paper with the term “3D Gaussians”.\n\n\n\n\n[1] Kocabas, Muhammed, et al. \"Hugs: Human gaussian splats.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2024.\n\n[2] Hu, Shoukang, Tao Hu, and Ziwei Liu. \"Gauhuman: Articulated gaussian splatting from monocular human videos.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dnxRgqu5IM", "reviewer": "Reviewer_Vqeb", "summary": "This paper proposes a method for reconstructing gaussian-based human avatars from occluded captures. The gaussian avatar model is based on GauHuman, which is optimized in multiple stages, including using a diffusion based prior in the canonical space to recover the complete human. In the first stage, a consistent human silhouette is generated using \"z-buffered\" pose key points. Next, this silhouette is filled in with textures using SDS which are then further refined by a diffusion model (same as one used for SDS) using the phrase - \"the same person standing in two different rooms\". Finally, these inpainted images are used to refine the color of the gaussians to generate the final avatar. Both quantitative and qualitative results demonstrate the effectiveness of the proposed method over prior work. While each component of the model is not novel in itself, the overall method is, and it works well.", "questions": "1) I'm curious as to whether the authors tried inpainting in the deformed space as well? If yes, does it work better or worse. It will be great if the authors could include those results as well", "limitations": "Yes", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The use of the diffusion model to generate the silhouette, inpaint it in the canonical space and use it as pseudo ground-truth is novel.\n2) The architecture is well motivated, with the purpose of each component very intuitive and well ablated.\n2) The evaluations seem thorough for the most part, apart from the ablations.\n3) The paper is well written.", "weaknesses": "1) In Figure 6, it would be nice to show results on both subjects for all experimental configurations (Exp A to E). In its current form it is incomplete.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for your time and the helpful comments! We address your concerns below.\n\n> Incompleteness of Figure 6\n\nThanks for the suggestion. The reason why we do not include results for all experiments for both subjects in Figure 6 is to showcase the benefits of our proposed components on the canonical and deformed spaces. We aim to show improvements in rendered appearance in the deformed space with the subject on the left, and aim to show how human completeness is improved and artifacts are reduced in the canonical space with the subject on the right. We include more visual experiment results in the rebuttal PDF.\n\n> Inpainting in deformed space\n\nIn the Optimization Stage, we apply Score Distillation Sampling (SDS) to both the deformed and canonical spaces (see Sec. 4.2 and Eq. 5 of the main paper). The space is controlled by a random variable with a 75% probability of applying SDS on the deformed space and 25% on the canonical space. We add additional experiments to show that our proposed random mix of deformed/canonical space SDS yields the best results compared to applying SDS solely on the deformed space or the canonical space (please see Fig. 4 in the rebuttal PDF). \n\nIn the Refinement Stage, we operate solely in the deformed space – we render posed human renderings from the Optimization Stage and juxtapose them with occluded partial observations from the input video in order to perform our in-context inpainting.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aSH2j3RY76", "reviewer": "Reviewer_WHSE", "summary": "This paper introduces OccFusion, a method for rendering occluded humans. \n\nSimilar to other 3DGS-based human rendering methods, OccFusion optimizes a set of 3D Gaussians to improve training and rendering speed. \n\nOccFusion proposes adopting generative diffusion priors to ensure complete and high-quality renderings to aid in reconstruction.\n\nOccFusion was evaluated on ZJU-MoCap and challenging OcMotion sequences, and it achieved state-of-the-art performance in rendering occluded humans.", "questions": "It is better to include more video results and add diversities of datasets.", "limitations": "The authors adequately addressed the limitations.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The 3-stage strategy is efficient in handling occluded humans. \n\nDuring the initialization stage, inpaint occluded human visibility masks into complete human occupancy masks. \n\nDuring the optimization stage, initialize a set of 3D Gaussians and optimize them by SDS in both the posed and canonical space. \n\nDuring the refinement stage, inpaint unobserved regions of the human with context from partial observations and renderings from the previous stage.\n\nThis method outperforms the state-of-the-art in rendering humans from occluded videos.", "weaknesses": "Lack of discussion about NeRF in the Wild, Ha-NeRF, Gaussian in the Wild and NeRF On-the-go.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive assessment of our work and the helpful comments! We address your concerns below.\n\n> Lack of discussion about NeRF in the Wild, Ha-NeRF, Gaussian in the Wild and NeRF On-the-go\n\nThanks for the suggestion. We will add the following sentences to section 2.2 of our Related work section in our final version:\n\n```\nNeRF-W [1] and other works [2,3,4] are able to account for photometric variation and transient occluders in complex in-the-wild scenes, allowing them to render consistent representations from unconstrained image collections. \nHowever, these works are not designed to handle dynamic objects like humans.\n```\n\n\n\n[1] Martin-Brualla, Ricardo, et al. \"Nerf in the wild: Neural radiance fields for unconstrained photo collections.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.\n\n[2] Chen, Xingyu, et al. \"Hallucinated neural radiance fields in the wild.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n[3] Ren, Weining, et al. \"NeRF On-the-go: Exploiting Uncertainty for Distractor-free NeRFs in the Wild.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[4] Zhang, Dongbin, et al. \"Gaussian in the Wild: 3D Gaussian Splatting for Unconstrained Image Collections.\" arXiv preprint arXiv:2403.15704 (2024).\n\n\n> More visual results\n\nWe provide more experiment results in the PDF of the general rebuttal, and we will include more extensive video results in the final version.\n\n> Diversity of dataset\n\nThis is a great observation. A big restriction of current human rendering datasets (ZJU-MoCap, OcMotion) is that they lack diversity. ZJU-MoCap consists of humans rotating in place in a brightly lit motion capture environment. While OcMotion is more representative of a real world scene, its diversity is still lacking, with all the sequences being collected in the same indoor room. In addition, since both datasets are collected from Chinese universities, the subjects are all East Asian men. We believe that a promising future step for this field is to collect more diverse data to test the generalizability of human rendering methods like ours. We will add a discussion of dataset diversity to our Limitations section in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IZpTYVnSMP", "reviewer": "Reviewer_BvJA", "summary": "This paper presents OccFusion, a 3D human avatar creation system that combines 3D Gaussian Splatting (3DGS) and 2D diffusion models to effectively render occluded regions. To this end, three stages are designed. First, in the initialization stage, complete human masks (without occluder) are obtained using a pre-trained diffusion model. Then, in the optimization stage, 3DGS is optimized with score distillation sampling (SDS) to recover occluded regions. Finally, in the refinement stage, in-context inpainting is designed to improve the rendering quality.", "questions": "1. What makes the result look so blurry even for visible areas?\n2. In Fig. 5, the second column and second row results do not have visible humans in the reference.\n3. How much of the mask (M) and complete human mask (\\hat{M}) are accurate? Also, how much of the optimization stage is robust to wrong M and \\hat{M}?\n4. Fig. 3 shows a pose-to-image generation example, while the actual use case is (image+pose+mask)-to-image as in Fig. 4. In the case of Fig. 4, the pose simplification of FIg. 3 is still necessary?\n5. Fig. 5 shows that the leftmost column is an input. What do the authors mean by input? As far as I understand, 3DGS is optimized to replicate images of a single person in the training set, and there is no input to the 3DGS. Or the 3DGS takes images of any person and can render images of that person with novel poses and viewpoints?", "limitations": "The major limitation is the rendering quality. Most of the rendered images are quite blurry.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "What the authors try to address (recovering 3D human avatar from human images with occlusions) is an important and unexplored problem. The authors’ design, which incorporates pre-trained diffusion models, is a reasonable choice.", "weaknesses": "1. Overall, the rendering quality is bad. This is applicable even for visible areas. Fig. 5 and 6 shows that the results are quite blurry especially on the OcMotion dataset.\n\n2. I don’t think the rightmost binary masks in Fig. 4 are consistent. All of them obviously have different pants silhouettes. Given these inconsistent masks, which masks did the authors use to optimize 3DGS? \n\n3. Overall, paper writing should be polished.\n\n--- post rebuttal ---\n\nThe authors addressed my concerns well. I'm raising my rating.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for your time and the helpful comments! We address your concerns below.\n\n> Rendering quality seems blurry\n\nWith only 10 mins of training time, OccFusion surpasses state-of-the-art occluded human rendering methods by a significant margin qualitatively and quantitatively, as illustrated in Table 1 and Fig. 1 and 5 of the main paper. With that said, with a longer training time, OccFusion is able to render in higher quality with much less blur. We added supplementary experiments to validate this statement – please see qualitative and quantitative results in Fig. 1 in the rebuttal PDF. We believe that our proposed 10-minute version of OccFusion is able to achieve the best balance of rendering quality and efficiency.\n\n> Is $M$ accurate?\n\nAs mentioned in the implementation details section of our Appendix, we derive $M$ from SAM (ICCV 2023), which is among the state-of-the-art in image segmentation, making it a very reliable way to calculate $M$. We would like to note that numerous other human rendering works [1, 2, 3] also use SAM as their go-to method for obtaining pseudo ground truth human masks during preprocessing.\n\n[1] Kocabas, Muhammed, et al. \"Hugs: Human gaussian splats.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2024.\n\n[2] Pang, Haokai, et al. \"Ash: Animatable gaussian splats for efficient and photoreal human rendering.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[3] Hu, Liangxiao, et al. \"Gaussianavatar: Towards realistic human avatar modeling from a single video via animatable 3d gaussians.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n\n> Binary masks do not seem consistent — Is $\\hat{M}$ accurate?\n\nWhen dealing with in-the-wild data, the ground truth silhouette of occluded regions is always unknown. Since the silhouette of the human across different frames is supposed to be inconsistent due to human (and sometimes garment) movement, we will never know with certainty what the changes in the ground truth silhouettes would be across a video sequence. However, we find that by using a pre-trained Stable Diffusion prior that has been trained on a great amount of similar data with enforced consistency, we can ensure that the inpainted $\\hat{M}$ falls into a reasonable estimation given the context of the pose and visible regions. As Fig. 4 of the main paper demonstrates, the shape of inpainted binary masks is much more reliable and consistent than the appearance of the corresponding RGB inpainted images. In our pipeline, we generate only one mask for every frame and find that $\\hat{M}$ does well in supervising the training of later stages.\n\n> Robustness of Optimization Stage to wrong $M$ and $\\hat{M}$\n\nAs we covered in the past two responses, the fact that we utilize pre-trained SAM to get $M$ and pose-conditioned pre-trained Stable Diffusion to inpaint $\\hat{M}$ makes getting a “wrong” mask extremely unlikely. However, to test the robustness of our method to inpainted masks, we add experiments on ZJU-MoCap supervised on the complete SAM masks obtained from the **unoccluded humans** in the optimization stage. Please see the qualitative and quantitative results in Fig. 2 in the rebuttal PDF. We find that using the inpainted $\\hat{M}$ can lead to good rendering quality that is comparable to using masks derived from the unoccluded images, validating that our model is robust to variances that exist in the masks.\n\n> Is pose simplification necessary for image + pose + mask -> image?\n\nThe purpose of Figure 3 is to show that Stable Diffusion fails when conditioned on challenging 2D poses. Since we use the same model for the image inpainting process (with the masks being used to calculate the necessary regions for inpainting), the pose simplification step we proposed is still necessary for the Stable Diffusion model to accurately condition on the pose of the human. So, we apply the pose simplification step whenever self-occluded joints are present to discourage problematic generations.\n\n> Fig. 5 shows that the leftmost column is an input. 3DGS is optimized to replicate images of a single person in the training set, and there is no input to the 3DGS.\n\nThis is correct. The leftmost columns are not really inputs but reference images at the corresponding time stamp from the training point of view. We apologize for this error and will correct the figure to clarify this point.\n\n> Fig 5 Second Column Second Row does not have a visible human in reference\n\nIn Figure 5, we validate renderings from novel views. The reference images on the right show the views from the new camera perspective that is provided by the dataset. For an in-the-wild dataset like OcMotion, sometimes the camera cannot capture the complete human due to occlusions. In the case of the example on row 2 column 2, the human is completely occluded from the reference camera view. We will make this point more clear in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank all of the reviewers for their thoughtful feedback and helpful suggestions! We agree that occluded human reconstruction from a monocular video is an “important and unexplored problem” (Reviewer BvJA) that is “relatively new and practical” (Reviewer gVDG). By proposing a “novel”, “well motivated”, and “intuitive” (Reviewer Vqeb) three-stage architecture composed of Initialization, Optimization, and Refinement stages, our method is not only “efficient” (Reviewer WHSE), but also “demonstrates superior performance compared to previous methods” (Reviewer 9Cmr). 

In the rebuttal, we clarify some concerns raised by the reviewers and, as requested, show some additional experiments to address the reviewers’ shared concerns:

> Effect of training time on rendering quality (Reviewer BvJA, gVDG, 9Cmr)

We explore the capacity of the models by training both OccFusion and OccGauHuman for 60 minutes, compared to the 10 minutes reported in the paper. After 60 minutes of training, both of the methods are fully converged. As shown in Fig. 1 in the rebuttal PDF, OccGauHuman is still unable to recover occluded regions of the human. However, training OccFusion for 60 minutes leads to less blurs in the final renderings. It also achieves slightly better quantitative performance, with a higher PSNR and lower LPIPS. Notably, with only 10 minutes of training, OccFusion is able to achieve on-par performances, yielding the best efficiency-performance trade off. 

> Robustness of training to ‘wrong’ masks (Reviewer BvJA)

For in-the-wild occluded videos, there are no ‘ground truth’ masks for the occluded body regions due to unknown human/garment deformations. Relying on the state-of-the-art pre-trained priors brought by the Segment Anything model (SAM) and Stable Diffusion, the segmented/inpainted masks are expected to be reasonable and coherent across frames. To test the robustness of our method to variances in the in-painted masks, we add comparison experiments on ZJU-MoCap that supervise using the complete SAM masks obtained from the **unoccluded humans** with minimum variances.  Please see the qualitative and quantitative results in Fig. 2 in the rebuttal PDF. We find that using the inpainted $\hat{M}$ leads to a good enough rendering quality comparable to using masks derived from the unoccluded images, validating the robustness of our model.


> Applying SDS on RGB vs on Human Occupancy Maps (Reviewer gVDG)

In Fig. 3 of the rebuttal PDF, we include additional experiments comparing the rendering results of applying SDS on RGB vs. on human occupancy maps (as proposed). It is clear that applying SDS on RGB leads to defective renderings as well as inferior quantitative results. This experiment validates our claim made in Sec. 4.1 and Fig. 4 in the main paper.

> Applying SDS on Canonical and Deformed Spaces Only vs. Jointly (Reviewer Vqeb)

In the Optimization Stage, we apply Score Distillation Sampling (SDS) to both the deformed and canonical spaces (see Sec. 4.2 and Eq. 5 of the main paper). The space is controlled by a random variable with a 75% probability of applying SDS on the deformed space and 25% on the canonical space. We add additional experiments to show that our proposed random mix of deformed/canonical space SDS yields the best results compared to applying SDS solely on the deformed space or the canonical space (please see Fig. 4 in the rebuttal PDF).

Meta-review: The paper received good scores. In the rebuttal phase, the authors engaged in fruitful discussions with the reviewers. Many of the smaller concerns were removed, and the reviewers agreed to recommend accepting the work. AC agrees. Congrats. 

Please include the additional clarification, as brought up during the rebuttal phase, while preparing the final version of the paper.
Decision: Accept (poster)