你好，以下是一篇单视角重建的文章，为什么会说3d生成的结果太well-defined呢？
GSD: View-Guided Gaussian Splatting Diffusion  for 3D Reconstruction  Yuxuan Mu1,2⋆ , Xinxin Zuo2 , Chuan Guo1,2 , Yilin Wang1,2, Juwei Lu2, Xiaofeng Wu2, Songcen Xu2, Peng Dai2, Youliang Yan2, and Li Cheng1  1 University of Alberta, Edmonton AB T6G 2R3, Canada {ymu3, <PRESIDIO_ANONYMIZED_EMAIL_ADDRESS> 2 Huawei Noah’s Ark Lab  Abstract. We present GSD, a diffusion model approach based on Gaussian Splatting (GS) representation for 3D object reconstruction from a single view. Prior works suffer from inconsistent 3D geometry or mediocre rendering quality due to improper representations. We take a step towards resolving these shortcomings by utilizing the recent state-of-theart 3D explicit representation, Gaussian Splatting, and an unconditional diffusion model. This model learns to generate 3D objects represented by sets of GS ellipsoids. With these strong generative 3D priors, though learning unconditionally, the diffusion model is ready for view-guided reconstruction without further model fine-tuning. This is achieved by propagating fine-grained 2D features through the efficient yet flexible splatting function and the guided denoising sampling process. In addition, a 2D diffusion model is further employed to enhance rendering fidelity, and improve reconstructed GS quality by polishing and reusing the rendered images. The final reconstructed objects explicitly come with high-quality 3D structure and texture, and can be efficiently rendered in arbitrary views. Experiments on the challenging real-world CO3D dataset demonstrate the superiority of our approach. Project page: https://yxmu.foo/GSD/  Keywords: Object Reconstruction, Gaussian Splatting, Guided Diffusion Model  1 Introduction  Given the abundance of image data in the real world, the problem of 3D reconstruction from single-view images has garnered notable attention [11, 41, 44, 49]. While humans can effortlessly deduce the general object shape and even imagine its texture from unseen views, for computational models, the problem becomes highly non-trivial. As will be described next, three key aspects underpin this problem. First, a proper 3D representation capable of encoding high-fidelity 3D information, while being compatible with various levels of quantization. Second, akin to the human perception system, it is crucial to have a generative model  ⋆ This work was done during an internship at Huawei Noah’s Ark Lab  arXiv:2407.04237v3 [cs.CV] 19 Jul 2024 GS ~ N(0, I) GS x0  GS Denoising  View-Guided Sampling  Fig. 1: A illustration of our View-Guided Gaussian Splatting Diffusion framework for single-view 3D reconstruction. It works by progressively denoising a randomly initialized set of Gaussian Splatting (GS) ellipsoids with continuous guidance from the discrepancies between the input and rendered images. The gray arrow represents the splatting-based GS rendering, while the orange arrow depicts the backpropagation of guidance gradients. The diffusion model built directly upon the GS representation in our context provides explicit geometry information. The view-guided sampling takes the advantage of splatting function to faithfully yet efficiently obtain fine-grained features from the given view.  being able to produce an object with diverse appearances of the object’s backside and being faithful to the input views. Finally, the ability to efficiently and precisely render a 3D object into an arbitrary view. Existing efforts often struggle to properly address one or multiple aspect(s) of the above three. For instance, the recent 2D novel view synthesis methods [3,18,21,31,42,52] usually fall short in maintaining 3D consistency. Another line of research [11, 12, 19, 20, 24, 41, 44, 45, 47] based on explicit 3D representations such as voxels, point clouds and meshes, are limited to coarse geometry and suboptimal rendering quality, though offering consistent 3D rendering. This may be attributed to the low resolution or sparse nature of 3D features (e.g., points, voxels) and the challenges of their discretization (e.g., mesh) in deep learning models. Implicit 3D representation, on the other hand, formulates 3D space as a query-based implicit function [1, 7, 25, 26, 33], achieving remarkable quality in single-view 3D reconstruction in well-defined canonical space [5, 15, 49]. Unfortunately, they typically rely on cumbersome efforts such as marching cubes for 3D geometry extraction and view rendering. Motivated by these observations, we introduce a novel framework, GSD, for high-quality single-view 3D reconstruction by building a generative Diffusion Transformer (DiT) [28] upon the emerging Gaussian Splatting (GS) representation [17]. Specifically, GS encodes a scene by a set of GS ellipsoids, with each ellipsoid parameterized by its center position, covariance, regional color, and opacity. Unlike existing 3D representations, GS explicitly encodes 3D geometry and texture in high resolution and density. Furthermore, due to its spatial explicitness, we can easily deploy a point-space DiT even without positional encoding [27]. Different from other diffusion models using classifier-free conditioning with image encoders [23,27,52], our GS DiT enables efficient fine-grained image conditioning through its unique splatting-based rendering and loss-guided sampling [35], which also ensures fidelity to the given-view images. Upon the GS representation, a category-specific 3D DiT is trained to capture the space of plausible 3D objects in terms of their diverse geometries and textures. As illustrated in Fig. 2, when w/o an input image, our diffusion model learns to generate high-fidelity 3D objects with distinct geometries and textures. When an input image is provided, the same diffusion model is used to reconstruct the specific 3D object, faithful when rendering to the same view. This process is presented in Fig. 1. During test time, the GS object at each denoising step is projected to the given view through the differentiable splatting-based rendering. The gradients of discrepancies between the rendered and the reference images are then backpropagated to the corresponding GS samples to refine the 3D object at the current step, similar to classifier guidance [35]. This approach is simple yet effective, and can be easily adapted to multi-view reconstruction. In addition, an auxiliary 2D diffusion model is employed to further improve the quality of rendered images, which reciprocally facilitates better 3D reconstruction. Our main contributions can be summarized as follows:  – Our proposed GSD is, to our knowledge, the first diffusion model that directly models raw GS representation capturing its 3D generative prior for single-view reconstruction. – The GS DiT intuitively comes with an effective yet flexible view-guided sampling strategy that can extract fine-grained features from the given views using the efficient splatting function. Given an input image at test-time, the guided iterative denoising of our GS-based diffusion model allows a progressive refinement of the reconstructed 3D object consistent with the input view. – Empirical experiments of the real-world CO3D dataset demonstrate the superiority of our approach when compared to the state-of-the-art. Our approach is flexible and can also work with multi-view images.