OccFusion: Rendering Occluded Humans with Generative Diffusion Priors

Authors: adam  sun
tiange  xiang
scott  delp
li  fei-fei
ehsan  adeli
occfusion  occfusion
occgauhuman  occgauhuman
zju-mocap  ocmotion
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 20:08:01

Abstract: Existing human rendering methods require every part of the human to be fully visible throughout the input video. However, this assumption does not hold in real-life settings where obstructions are common, resulting in only partial visibility of the human. Considering this, we present OccFusion, an approach that utilizes efficient 3D Gaussian splatting supervised by pretrained 2D diffusion models for efficient and high-fidelity human rendering. We propose a pipeline consisting of three stages. In the Initialization stage, complete human masks are generated from partial visibility masks. In the Optimization stage, 3D human Gaussians are optimized with additional supervisions by Score-Distillation Sampling (SDS) to create a complete geometry of the human. Finally, in the Refinement stage, in-context inpainting is designed to further improve rendering quality on the less observed human body parts. We evaluate OccFusion on ZJU-MoCap and challenging OcMotion sequences and found that it achieves state-of-the-art performance in the rendering of occluded humans.

Main paper: 
 introduction: 
 Rendering 3D humans from monocular in-the-wild videos has been a persistent challenge, with significant implications in virtual/augmented reality, healthcare, and sports. Given a video of a human and use Linear Blend Skinning (LBS) to transform the human into the posed space. Gaussian splatting methods achieve state-of-the-art performance of dynamic humans with fast training times and real-time rendering, causing them to be the more desired method [46, 14] .Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: Our work proposes a novel three-stage pipeline for rendering occluded humans with Gaussian splatting and diffusion priors that outperforms the state-of-the-art in the field. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
occluded human rendering: 
 Reconstructing complex scenes in the wild is a well-studied problem. NeRF-W [36] and other works [3, 47, 75] are able to account for photometric variation and transient occluders, allowing them to render consistent representations from unconstrained image collections. However, these works are not designed to handle dynamic objects like humans.Rendering humans in occluded settings, on the other hand, is relatively understudied. Sun et al. [51] utilize a layer-wise scene decoupling model to decouple humans from occluding objects. OccNeRF [64] combines geometric and visibility priors with surface-based rendering to train a human NeRF model, while Wild2Avatar [63] proposes an occlusion-aware scene parametrization scheme to decouple the human from the background and occlusions. While these works provide decent renderings of humans free of occlusions, they are slow and impractical due to their usage of NeRFs. A concurrent work to ours is OccGaussian [67] , which also proposes to model occluded humans with 3D Gaussians by performing an occlusion feature query in occluded regions. We provide comparisons to their published results in Table 1 .Inferring the appearance of unobserved regions of 3D scenes requires the usage of generative models. The recent success of 2D diffusion models has made them the preferred model to use for generation [54, 21, 34, 48, 32] . To lift 2D diffusion models for 3D content generation, DreamFusion [45] proposed Score Distillation Sampling (SDS), a commonly used method for utilizing a pre-trained 2D diffusion model to supervise 3D content generation [30, 70, 53, 55] .Diffusion models can also be used as priors for training NeRFs and Gaussian splatting, combining reconstruction with generation [61, 78, 79, 62, 66, 73] . ReconFusion [61] uses SDS in conjunction with multi-view conditioning to synthesize the appearance of unobserved regions of a scene from sparse views, while BAGS [78] utilizes SDS to supervise a Gaussian splatting model.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: Our paper includes a "Limitations" section in Section 6 that acknowledges the inconsistency of generative diffusion models, which are central to our approach. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
preliminaries: 
 Before introducing our method, we provide an overview of key fundamentals in 3D human modeling using SMPL (subsection 3.1). Then, we discuss 3D Gaussian splatting, and how it can be applied to human modeling (subsection 3.2). Finally, we propose OccGauHuman, a simple improvement of GauHuman [14] that is better designed for occluded human rendering (subsection 3.3).SMPL [35] is a model that parametrizes the human body with a 3D surface mesh. To transform between the canonical space to a pose space, the Linear Blend Skinning (LBS) algorithm is used. Given a 3D point x c in the canonical space and the shape β and pose θ parameters of the human, a point in the posed space can be calculated as:EQUATIONwhere J contains K joint locations, G k and b k are the transformation matrix and translation vector, and w k ∈ [0, 1] are a set of skinning weights. The SMPL representation is commonly used as a geometric prior for human rendering [64, 63, 46, 14, 57, 19, 72] .Rendering with 3D Gaussian Splatting 3D Gaussian splatting. 3D Gaussian splatting [24] models a scene as a set of 3D Gaussians Π. Each Gaussian is defined by its 3D location p i , opacity o i ∈ [0, 1], center µ i , covariance matrix Σ i , and spherical harmonic coefficients. The i-th Gaussian is defined as o i e -1 2 (p-µi) T Σ -1 i (p-µi) . During rendering, these 3D Gaussians are mapped from the 3D world space and projected to the 2D image space via α-blending, with the color of each pixel being calculated across the N 3D Gaussians as:EQUATIONwhere c j is the color and α is the z-depth ordered opacity. During the training process, 3D Gaussians are adaptively controlled via densification (splitting and cloning) and pruning until they achieve the optimal density to adequately represent the scene.GauHuman [14] . In the line of work that uses 3D Gaussian splatting for human rendering [46, 26, 29, 12] , GauHuman is a representative approach due to its balance between efficiency and rendering quality. After initializing 3D Gaussians on the vertices of the SMPL mesh, GauHuman learns a representation of the human in canonical space and utilizes LBS to transform each individual Gaussian into the posed space. A pose refinement module M LP Φpose and an LBS weight field module M LP Φ lbs are used to learn the LBS transformation, and a merge operation based on KL divergence is used along with splitting, cloning, and pruning to help the 3D Gaussians reach convergence.We base our method on GauHuman due to its fast training and state-of-the-art representative ability. GauHuman's code is distributed under the S-Lab license and can be accessed here.In common human rendering tasks, videos are captured in a clean environment, with every pixel in the image belonging to either the human or the background. By using a semantic segmentation model such as SAM [25] to preprocess a video, we can train the human rendering model only on pixels labeled as "human". However, occlusions in the videos may lead to sparse observations of the human. As a result, fitting NeRF-based human rendering models on only the visible human pixels results in an incomplete geometry with lots of artifacts [64, 63] .Gaussian splatting-based rendering models [24] are especially suitable for human modeling tasks due to their explicit geometry and point-based representation. In this section, we present three straightforward tweaks of GauHuman [14] to make it perform better on videos with occlusions: (1) Firstly, as discussed above, we train the model on visible human pixels only, ensuring that occlusions do not result in learned sparsity on the human model. (2) We adjust the loss weights to put more weight on the mask loss computed between rendered human occupancy maps and the segmentation masks -we found that this helps learn more crisp human boundaries. (3) We disable the densification and pruning of 3D Gaussians during training -this helps maintain a rather complete human geometry based on the SMPL initialization.The resulting OccGauHuman model serves as an improved baseline for occluded human reconstruction and as a starting point for our method. Benefits brought by our updates compared to the original GauHuman are presented in Table 1 , as well as in Figure 7 .Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [No] Justification: We are not including code in our submission. However, we provide implementation details in our supplementary materials. Guidelines:• The answer NA means that paper does not include experiments requiring code. • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
occfusion: 
 In our approach, we train a Gaussian splatting-based human rendering model on the visible pixels of a human. However, recovering occluded content for a dynamically moving human is not trivial -humans are usually in challenging poses, and complex occlusions can cause additional issues. It is also essential to preserve a consistent human appearance and geometry across different frames.Considering these challenges, we propose our method OccFusion in multiple separate stages. In the Initialization stage (section 4.1), we inpaint occluded binary human masks for more reliable geometric guidance. In the Optimization stage (section 4.2), we use the inpainted masks to train a human rendering model based on GauHuman [14] while using Score Distillation Sampling (SDS) constraints on both the posed space and canonical space. In the Refinement stage (section 4.3), we fine-tune the trained model from the Optimization Stage with in-context inpainting to further refine the appearance of the human. An overview of our OccFusion is shown in Figure 2 . Generative diffusion models [48] have demonstrated promise to be used as priors for different tasks [22, 53] . The most straightforward method is to utilize a precomputed segmentation prior M and pose prior P to condition Φ [39, 76] to inpaint 1 -M -the image regions that are not occupied by the human. However, there are two significant barriers to such a straightforward approach. Conditioned human generation cannot handle challenging poses. It is true that a conditioned diffusion prior Φ is able to generate detailed images while staying consistent with the condition. However, since diffusion models like Φ are usually overfitted on more commonly seen poses, Φ usually fails to generate reasonable images when conditioned on challenging poses (see Figure 3 middle column). We attribute this limitation to the inappropriate 2D representation of P -when joints occlude each other, it is impossible to tell which joints are closest to the camera when they are projected to 2D. So, we propose to simplify the 2D representation of P. We apply a Z-buffer test on the depth map rendered from the SMPL mesh [35] and then calculate the distance d between its z-axis location and the corresponding 2D z-buffer. Given a pre-defined threshold σ, we deem a joint is self-occluded if d > σ. Self-occluded joints are ignored when projecting 3D joints onto the 2D canvas for conditioning Φ (see Figure 3 right column). Our simplification improves the generation quality of Φ for challenging poses.Per-frame inpainting cannot guarantee cross-frame consistency. Compared to image generation models, video generation models [11, 60, 7] are less accessible and much more expensive to run. Without an explicit modeling of object motion in the video, frame-by-frame generation with an image generative model leads to cross-frame inconsistency, which is not desirable for human reconstruction (see Figure 4 middle column). Instead of inpainting the occluded parts of the human directly with Φ, we claim that it is more feasible to inpaint binary human masks since small variations in the human silhouette are more acceptable (see Figure 4 right column). We first inpaint the RGB image I and then rely on an off-the-shelf segmentation model [23] to obtain the inpainted binary human masks { M}, which is used to assist the training of the rendering model in subsequent stages.After obtaining the inpainted masks { M} that outline a reasonable human silhouette, we build a Gaussian splatting model similar to the one described in section 3.2 for human rendering. The 3D Gaussians Π are initiated as the SMPL mesh vertices, which are able to be deformed to adapt to different poses through SMPL-based LBS (Equation 3.1). With the help of { M}, the training of Π consists of multiple photometric loss terms L photo :EQUATIONwhere L 1 is the L-1 loss, L 2 is the L-2 loss, SSIM(•) is the SSIM function [56] , LPIPS is the VGGbased perceptual loss [77] , I ′ is the rendered image from Π, and A is the rendered human occupancy map. Each of the loss terms is scaled by a weight hyperparameter λ.Even with the supervision of { M}, geometry inconsistency still exists. Although inconsistent human masks affect the training of Π much less than inconsistent images, human completeness cannot be guaranteed without further steps.Using diffusion priors to enforce human completeness. We build off of the insights from [53, 59, 70] and apply Score Distillation Sampling (SDS) [45] to improve the quality of human renderings and reduce artifacts. Instead of applying SDS on RGB images I ′ , which causes appearance inconsistency, we apply it directly to the rendered human occupancy maps A so that diffusion scores are propagated to encourage complete A:EQUATION)where t is a scheduled time stamp, w(•) is a weighting function, ϵ(•) is the UNet noise estimator in Φ, and ϵ is the injected Gaussian noise.Using diffusion priors to regularize canonical pose. In-the-wild videos often involve very sparse observations of the human, with only incomplete regions of the human visible in each frame. To further enforce completeness, we propose to render the human in the canonical Da-pose P with the human oriented at a random angle ∈ {k π 9 , k ∈ Z}. Applying SDS on the canonical renderings serves as regularization and is randomly activated during training. Overall, at each training step in the Optimization Stage, the 3D Gaussians Π are optimized towards:EQUATIONwhere ρ is a random variable that has a 75% chance to be 1 and 0 otherwise. The Optimization stage results in a complete and coherent geometry regardless of the viewing angle. As shown in Figure 6 Exp. C and D, applying diffusion priors on rendered human occupancy maps is not able to recover the missing appearances of the human. This motivates the need for a subsequent stage that keeps refining Π for better appearance.The refinement of the appearance of 3D objects is not a new topic [53, 31, 70] . However, no existing generative models are capable of handling the consistency of appearance of a human across different frames and poses. We attribute this difficulty to the denoising process used in generative priorsrandom noise is injected to rendering at each SDS step which leads to uncertain results. This is infeasible for reconstruction tasks, which require frame-consistent representations that agree with all observations. Our approach focuses on generating inpainted images of the occluded human offline to use as references. We first identify the occluded regions to be inpainted R by using the rendered human occupancy masks A from the Optimization Stage and pre-computed human visibility masks M: R = (1 -M) • A. In order to encourage the generated regions to be more consistent with the partial observations, we propose in-context references inspired by in-context learning in language models [1] . Although renderings from the Optimization Stage lack sharp and high-fidelity details, they resemble complete human geometries and possess good enough features that can be used as a coarse reference to guide Φ to inpaint similar contents at occluded body regions. To achieve this, we stack Î and I together as a single image input to Φ with an additional prompt phrase -"the same person standing in two different rooms".We use the inpainted RGB images { Ĩ} along with other priors to finetune Π via photometric losses. Since diffusion models still tend to be somewhat inconsistent, we smooth training by putting more weight on perceptual loss terms and use L1 loss for the pixel-wise loss terms for its high robustness to variance:∇ Π λ rgb L 1 (M • I, M • I ′ ) + λ mask L 2 ( M, A) + λ gen L 1 ( Ĩ, R • I ′ ) + λ lpips LPIPS(I, I ′ ) . (6)We train our entire pipeline for only 10 minutes on a single TITAN RTX GPU. More implementation details are provided in supplementary materials. 
experiments: 
 In this section, we conduct quantitative and qualitative evaluation of our approach against state-of-theart methods. Then, we conduct ablation studies of our entire pipeline, demonstrating that each stage is necessary for optimal performance. More experiments and results can be found in supplementary materials. Figure 5 : Qualitative comparisons on simulated occlusions in the ZJU-MoCap dataset [44] (left column) and real-world occlusions in the OcMotion dataset [16] . ON denotes OccNeRF [64] and OGH denotes OccGauHuman.ZJUMoCap. ZJU-MoCap [44] is a dataset consisting of 6 dynamic humans captured with a synchronized multi-camera system. Since the humans are in a lab environment free of occlusions, we follow OccNeRF's [64] protocol to simulate occlusion of the human, masking out the center 50% of the human pixels for the first 80 % of frames. To challenge OccFusion on videos with even sparser frames, we use only 100 frames from the first camera with a sampling rate of 5 to train the models and use the other 22 cameras for evaluation.OcMotion. OcMotion [15] comprises of 48 videos of humans interacting with real objects in indoor environments. Experiments are conducted on the same 6 sequences adopted by Wild2Avatar [63] , which are selected to provide a diverse coverage of real-world occlusions. We form sparser subsequences by sampling only 50 frames from each sequence to train the models. Evaluation. We compare our OccFusion to OccNeRF [64] , OccGaussian [67] , and Wild2Avatar [63] , the state-of-the-art in occluded human rendering. We also compare our results to GauHuman [14] , HumanNeRF [57] , and 3DGS-Avatar [46] , popular human rendering methods not designed for occlusion. For fairness of comparison, all methods use the same set of segmentation masks and pose priors. We train GauHuman and OccGauHuman for 10 minutes each. We evaluate the methods both quantitatively and qualitatively. For our quantitative evaluations, we calculate the Peak Signal-to-Noise Ratio (PSNR), Structural SIMilarity (SSIM), and Learned Perceptual Image Patch Similarity (LPIPS) metrics against the ground truth images. Since no ground truth is provided for OcMotion, we calculate the metrics on visible pixels only. For qualitative evaluations, we render the human from novel views and assess the quality of the renderings.We provide quantitative metrics averaged over all the sequences in Table 1 . Overall, methods designed for occluded human rendering tend to outperform their traditional counterparts. Among those methods, OccFusion consistently performs up to par or better than the state-of-the-art on both datasets while significantly beating all the baselines on LPIPS.Qualitative results on novel view synthesis can be found in Figure 5 . OccNeRF [64] has trouble generating unseen regions and renders significant discoloration and floaters when faced with occlusion.On the other hand, OccGauHuman's renderings are blurry and occasionally incomplete. We observe that OccFusion is the only method to consistently render sharp and high-quality renderings free of occlusions.Ablation studies. We study the effect of each of our proposed components by adding them one by one and report average metrics on ZJU-MoCap in Table 2 . Each stage plays a part towards optimal performance. Qualitative results on our ablations are included in Figure 6 . We can see that the Initialization Stage helps enforce completeness for the initially incomplete human. The SDS regularization provided in the Optimization Stage helps remove floaters and artifacts in the posed and canonical space, further improving the shape of the human and enforcing completeness of the body. Finally, the Refinement Stage helps make the renderings more detailed in less observed regions, improving the rendering quality and greatly reducing the LPIPS. Does the proposed OccGauHuman perform better than GauHuman [14] in rendering occluded humans? In section 3.3, we present a simple upgrade for the state-of-the-art 3DGS based human rendering model GauHuman [14] to help it better handle occlusions. Our improvements are straightforward but effective. We show quantitative results in Figure 1 (Left) and Table 1 . As shown in Figure 7 , our improved OccGauHuman reconstructs a more complete human body than the vanilla GauHuman. Figure 7 : Qualitative comparisons on simulated occlusions in the ZJU-MoCap dataset [44] (left column) and real-world occlusions in the OcMotion dataset [16] (right column). GH denotes GauHuman [14] and OGH denotes OccGauHuman. 
discussions and conclusion: 
 Limitations. Recovering occluded dynamic humans is challenging. As mentioned in section 4.3, reconstructing a 3D human requires adhering to multiple consistencies. However, even with the state-of-the-art generative models, it is still impossible to perfectly maintain those consistencies for 4D content (3D + motion) generation. Although our proposed methods are specifically designed to eliminate potential variances when using generative priors, we can still observe some generations are less coherent (e.g. Figure 4 and Figure 8 ), which may hurt the training of the rendering model on all stages. Moreover, we found that conditioning generative models with 2D poses is weak -the pose of the generated human does not always align with the condition pose, which may introduce even more uncertainty for training. In future work, we hope to train our own consistency-aware diffusion model specifically finetuned on human data.Societal Impacts. Being able to reconstruct a human from an occluded monocular video can have a great societal impact. For example, having a high-fidelity 3D reconstruction of a human can help telemedicine practitioners become more immersed in the 3D space. While our research could lead to privacy concerns if humans are reconstructed without their consent, we believe that the benefits can be harnessed responsibly with appropriate safeguards. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: We mention the hardware used to train our method at the end of Section 4.3.We also break down the amount of time needed to train each stage in Table 2 .Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: We include discussion of societal impacts in 6.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our method utilizes per-video optimization, so no pretrained weights are provided and no models or data are released.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 
licenses for existing assets: 
 Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: In Subsec 3.3, we cite GauHuman in our paper and the license used, and provide a link to the code.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: New assets are not used or included in our submission.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: Our work does not involve crowdsourcing or research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: Our work does not involve crowdsourcing or research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: In this work, we propose OccFusion, one of the first works that utilize 3D Gaussian splatting for occluded human rendering. Our approach consists of three stages: the Initialization, Optimization, and Refinement stages. By combining the efficiency and representative ability of 3D Gaussian splatting with the generation capabilities of diffusion priors, our method achieves state-of-the-art in occluded human rendering quality as measured by the PSNR, SSIM, and LPIPS metrics while only taking around 10 minutes to train. We hope our work inspires further exploration into the capabilities of diffusion priors to aid in human reconstruction.For notation simplicity, we adopted alphabetic symbols in this paper to represent essential components in our framework. For better symbol-name correspondences, here we justify the implications of all symbols used in the paper in Table 3 . OccFusion requires several priors. We run SAM [25] to get all the human masks {M}. While we follow previous work [64, 63] and use the ground truth poses provided by ZJU-MoCap and OcMotion, pose priors P can be obtained via occlusion-robust SMPL prediction/optimization methods such as HMR 2.0 [8] and SLAHMR [69] for in-the-wild videos. Improving the quality of priors is not the focus of this work. We use the pre-trained Stable Diffusion 1.5 model [48] with ControlNet [76] plugins for SDS in all the stages.In the Initialization Stage, instead of inpainting incomplete human masks directly, we run the pretrained diffusion model to inpaint RGB images with 10 inference steps and 1.0 ControlNet conditioning scale. We use the positive prompt -"clean background, high contrast to the background, a person only, plain clothes, simple clothes, natural body, natural limbs, no texts, no overlay" and the negative prompt -""multiple objects, occlusions, complex pattern, fancy clothes, longbody, lowres, bad anatomy, bad hands, bad feet, missing fingers, cropped, worst quality, low quality, blurry". After inpainting the RGB images, we then run SAM-HQ [23] with P as the prompts to get { M}.In the Optimization Stage, we train the 3D human Gaussian Π from scratch by following the objective Equation 5. We set λ rgb = 1e 4 , λ mask = 2e 4 , λ ssim = 1e 3 , and λ lpips = 1e 3 . At each training step, we random switch the SDS regularization on either posed human space or the canonical Da-pose space with a probability of 75% and 25%. When applying SDS regularization on the canonical human space, we randomly rotate the human horizontally with a uniformly sampled degree in {k π 9 , k ∈ Z}. We set the SDS loss weights as λ pose = 2e 5 and λ can = 2e 5 . In this stage, we train Π for 1200 steps.In the Refinement Stage, we first generate the RGB human inpaintings via the proposed in-context inpainting method. We run the pretrained diffusion model with conditions on M, 10 inference steps, and 0.3 ControlNet conditioning scale. We did not use positive prompts for the inpainting but used the same negative prompts as in the Optimization Stage. During training, we set the loss weights as λ rgb = 1 and λ mask = 0.1, λ gen = 0.1, and λ lpips = 0.2. In this stage, we finetune Π for another 1800 steps with Gaussian densification and pruning enabled for the first 1000 steps.Effectiveness of in-context inpainting. We provide comparisons of the human in the Refinement Stage with and without in-context inpainting and provide qualitative comparisons in Figure 8 . While renderings from the Optimization stage are less detailed in occluded areas, our proposed in-context inpainting is able to generate the missing content and greatly increase the rendering quality in these areas. Applying SDS on RGB vs on Human Occupancy Maps. We include additional experiments comparing the rendering results of applying SDS on RGB vs. on human occupancy maps (as proposed). It is clear that applying SDS on RGB leads to defective renderings as well as inferior quantitative results. This experiment validates our claim made in Sec. 4.1 and Fig. 4 in the main paper.Robustness of training to inpainted masks. For in-the-wild occluded videos, there are no ground truth masks for the occluded body regions due to unknown human/garment deformations. Relying on the state-of-the-art pre-trained priors brought by the Segment Anything model (SAM) [25] and Stable Diffusion [48] , the segmented/inpainted masks are expected to be reasonable and coherent across frames. To test the robustness of our method to variances in the in-painted masks, we add comparison experiments on ZJU-MoCap that supervise using the complete SAM masks obtained from the unoccluded humans with minimum variances. Please see the qualitative results in Figure 10 . We find that using the inpainted M leads to a good enough rendering quality comparable to using masks derived from the unoccluded images, validating the robustness of our model.Can existing generative models recover an occluded human? While there are works for using generative diffusion models to render 3D humans conditioned on single [58] and multiple [49] images, none are able to condition on a monocular video of the person.Since [58] has not released code, we include results from InstantMesh [65] . We use the provided segmentation mask to mask the least occluded frame onto a white background and use it as conditioning.Novel view synthesis results are included in Figure 11 . InstantMesh is unable to recover a complete human geometry and fails to generate a reasonable appearance from the single image.For a more comprehensive presentation of the results, we include video renderings on all the training frames for both datasets. NeurIPS Paper ChecklistJustification: The authors have reviewed the NeurIPS Code of Ethics and strongly believe it conforms to the Code.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 


Decision: The paper received good scores. In the rebuttal phase, the authors engaged in fruitful discussions with the reviewers. Many of the smaller concerns were removed, and the reviewers agreed to recommend accepting the work. AC agrees. Congrats. 

Please include the additional clarification, as brought up during the rebuttal phase, while preparing the final version of the paper.

Reviews: review: paper_summary: paper_summary: This work presents a Gaussian-based approach to reconstruct 3D human poses from occluded monocular video. Building upon the success of previous methods like SDS, the paper suggests introducing a pre-trained diffusion prior to complete the occluded areas and further refine the missing appearance. The framework is evaluated on public datasets, including ZJU-MoCap and OcMotion, and shows improved efficiency and appearance quality.
strength_weakness: strength_weakness: - The ablation study could be more extensive. The authors mention that "applying SDS on RGB images causes appearance inconsistency" in line 192. It would be helpful if the authors could provide some visualizations to support this claim.
- If the optimization step can generate consistent appearances, the refinement stage could also introduce inconsistencies in the occluded areas, as there is no cross-frame constraint during the inpainting process.
questions: questions: - The supplementary videos lack method names, making it difficult to identify the final result of the proposed "OccFusion" method. It is unclear why some blur exists in the visible areas, as evident in cases like "ocmotion_0011_1" and "ocmotion_0011_2".
limitations: limitations: No negative societal impact.
paper_summary: paper_summary: OccFusion proposes an approach to model human bodies that fail under occlusion in monocular videos. The authors utilize 3D Gaussian splatting for efficient rendering and leverage pretrained off-the-shelf image diffusion models as 2D priors. Their approach involves a three-stage training process, sequentially refining estimates to achieve precise 3D human representations.  They demonstrate superior performance compared to previous methods, enhancing one of the baselines to better handle occlusion scenarios.
strength_weakness: strength_weakness: 1. Some important details are either omitted or mentioned too late in the text. Below are some examples:

1.1. While the extensive use of off-the-shelf models is well justified, it is only superficially discussed in the corresponding technical sections. For instance, the introduction does not mention the use of image generative models at the initialization stage to improve the masks. This is a core aspect of the work, and mentioning models like DreamFusion, Stable Diffusion, and ControlNet should be done when discussing the three stages, preferably in the introduction.

1.2. Figure 2 (method overview) is cumbersome and not very instructive. It does not clarify the three-stage training, and its understanding relies heavily on the detailed ablation study. I recommend reorganizing Figure 2 to include all off-the-shelf models used in the training pipeline. Even a plain list of inputs and outputs with the side models used at each stage would be more descriptive than the current Figure 2 and the introductory motivation.


2. Obscure training details

2.1 Training time: authors insist throughout the text that the training time of their method is 10 minutes. Assuming the monocular video is the only input, the result must depend on the length of the video, yet it is never discussed. 

2.2 It is unclear why 10 minutes is enough for good performance. Would estimates improve if trained longer?


3. Results

3.1 In Fig.6 Exp.A authors demonstrate that OccGauHuman estimate has some bodyparts ripped apart. This should not happen if initial splats are distributed all over SMPL mesh and "densification and pruning of splats is disabled. (line 137)". How can authors explain this behavior? 

3.2 In Table 2 GauHuman and OccGauHuman are trained for 10 and 2 minutes, correspondingly. This is unclear how estimates improve when OccGauHuman is trained for 10 minutes, as GauHuman. 
3.3 Continuing the issue 3.2, if all experiments with OccGauHuman model are also taken with 2-min limit, this makes the quantitative comparison unfair. maybe it is simply undertrained.


4. (minor) OccGauHuman. In Sec.3 authors should say explicitly that this method they propose is not the part of the contribution but the way to build a better baseline and a starting point for their own method.

5. (minor) Throughout the paper authors say "gaussian" referring to the splat or blob. It is better to avoid such namedropping, since Gaussian is a very broad term that can imply many different things and is itself an adjective and does not fit in the sentence properly.
questions: questions: I would like authors to answer to the issues I raise in Weaknesses. If I find them constructive, I would raise the rating.
limitations: limitations: -
paper_summary: paper_summary: This paper proposes a method for reconstructing gaussian-based human avatars from occluded captures. The gaussian avatar model is based on GauHuman, which is optimized in multiple stages, including using a diffusion based prior in the canonical space to recover the complete human. In the first stage, a consistent human silhouette is generated using "z-buffered" pose key points. Next, this silhouette is filled in with textures using SDS which are then further refined by a diffusion model (same as one used for SDS) using the phrase - "the same person standing in two different rooms". Finally, these inpainted images are used to refine the color of the gaussians to generate the final avatar. Both quantitative and qualitative results demonstrate the effectiveness of the proposed method over prior work. While each component of the model is not novel in itself, the overall method is, and it works well.
strength_weakness: strength_weakness: 1) In Figure 6, it would be nice to show results on both subjects for all experimental configurations (Exp A to E). In its current form it is incomplete.
questions: questions: 1) I'm curious as to whether the authors tried inpainting in the deformed space as well? If yes, does it work better or worse. It will be great if the authors could include those results as well
limitations: limitations: Yes
paper_summary: paper_summary: This paper introduces OccFusion, a method for rendering occluded humans. 

Similar to other 3DGS-based human rendering methods, OccFusion optimizes a set of 3D Gaussians to improve training and rendering speed. 

OccFusion proposes adopting generative diffusion priors to ensure complete and high-quality renderings to aid in reconstruction.

OccFusion was evaluated on ZJU-MoCap and challenging OcMotion sequences, and it achieved state-of-the-art performance in rendering occluded humans.
strength_weakness: strength_weakness: Lack of discussion about NeRF in the Wild, Ha-NeRF, Gaussian in the Wild and NeRF On-the-go.
questions: questions: It is better to include more video results and add diversities of datasets.
limitations: limitations: The authors adequately addressed the limitations.
paper_summary: paper_summary: This paper presents OccFusion, a 3D human avatar creation system that combines 3D Gaussian Splatting (3DGS) and 2D diffusion models to effectively render occluded regions. To this end, three stages are designed. First, in the initialization stage, complete human masks (without occluder) are obtained using a pre-trained diffusion model. Then, in the optimization stage, 3DGS is optimized with score distillation sampling (SDS) to recover occluded regions. Finally, in the refinement stage, in-context inpainting is designed to improve the rendering quality.
strength_weakness: strength_weakness: 1. Overall, the rendering quality is bad. This is applicable even for visible areas. Fig. 5 and 6 shows that the results are quite blurry especially on the OcMotion dataset.

2. I don’t think the rightmost binary masks in Fig. 4 are consistent. All of them obviously have different pants silhouettes. Given these inconsistent masks, which masks did the authors use to optimize 3DGS? 

3. Overall, paper writing should be polished.

--- post rebuttal ---

The authors addressed my concerns well. I'm raising my rating.
questions: questions: 1. What makes the result look so blurry even for visible areas?
2. In Fig. 5, the second column and second row results do not have visible humans in the reference.
3. How much of the mask (M) and complete human mask (\hat{M}) are accurate? Also, how much of the optimization stage is robust to wrong M and \hat{M}?
4. Fig. 3 shows a pose-to-image generation example, while the actual use case is (image+pose+mask)-to-image as in Fig. 4. In the case of Fig. 4, the pose simplification of FIg. 3 is still necessary?
5. Fig. 5 shows that the leftmost column is an input. What do the authors mean by input? As far as I understand, 3DGS is optimized to replicate images of a single person in the training set, and there is no input to the 3DGS. Or the 3DGS takes images of any person and can render images of that person with novel poses and viewpoints?
limitations: limitations: The major limitation is the rendering quality. Most of the rendered images are quite blurry.

Comments: title: Reply to Reviewer's Comment
Reply to Rebuttal
comment: We would like to thank the reviewer once again for your helpful feedback and suggestions. We hope that our additional experiments and responses have addressed your questions. We would be happy to continue the discussion if the reviewer has additional concerns.
We thank the reviewer for your continued feedback! We are glad that we were able to resolve most of your concerns with our method.

We would like to reiterate that with 10 minutes of training time, OccFusion significantly surpasses the previous state-of-the-art for occluded human rendering. In addition, for unoccluded scenes, OccFusion performs consistent with GauHuman, among the state-of-the-art in Gaussian-splatting based human rendering.

We understand that 60 minutes for a GS is relatively time consuming. So, for the camera-ready version, we will focus on presenting the 10-minute version, as it offers a strong balance between efficiency and rendering quality while still outperforming existing methods. 

We would like to once again express our heartfelt thanks for your helpful feedback and suggestions. We would be happy to continue the discussion if the reviewer has additional questions or concerns.
Thanks for the author's rebuttal. They addressed most of my concerns.

However, I share similar reservations with reviewer-BvJA regarding the rendering quality. As illustrated in the attached PDF file, extended optimization times reduce the blurs but introduce new artifacts, aliasing along the edges. In addition, a fitting time of 60 minutes for a GS is relatively time-consuming.  

Given the feedback from other reviewer and my ongoing concern, I will maintain my current score of 5.
Thank you for your thorough review and for considering our rebuttal.  

We would like to clarify that OccFusion employs the same rendering technique as introduced in GauHuman (CVPR 2024). In scenarios where there are no occlusions in the video sequence, OccFusion functions similarly to GauHuman, with the only addition being the opacity SDS in the Optimization stage.

As per your request, we conducted additional experiments to assess the capacity of OccFusion for rendering high-fidelity details. **Although figures cannot be included in this response**, we present the following quantitative comparisons after training both OccFusion and GauHuman for 10 minutes on a black background:

| Method       | PSNR  | SSIM   | LPIPS  |
|--------------|-------|--------|--------|
| GauHuman     | **31.44** | **0.9650** | 30.48  |
| **OccFusion**| 31.29 | 0.9627 | **30.40** |

Both quantitative and qualitative results indicate that OccFusion performs on par with GauHuman, with even better LPIPS, which often indicates of perceptual quality of renderings and **sharpness of textures**.

We appreciate the reviewer’s concern and will include both the quantitative and qualitative results in the camera-ready version.
Thanks for the rebuttal. It addressed my concerns well.
Thanks for your rebuttal. How the results become if the authors just use ZJU-Mocap as it is without simulating the occlusions? I want to know the rendering capability of the proposed method if there is no occlusion as the updated results (from longer training) still do not have sharp textures.
We thank the reviewer very much for your positive feedback on our rebuttal! We are glad that that we have effectively addressed the concerns you raised in your initial review,

Given that you mentioned in your original review that you would consider raising your score if our rebuttal was constructive, we would like to kindly ask the reviewer to reconsider their score. We would also be happy to engage further during the discussion period.
I thank authors for their instructive and well detailed answer. All of the issues I raised are well addressed.