以下是一篇单视角重建的文章，请问和单视角生成wonder3d有什么区别，有什么可借鉴的？
wonder3d
Abstract  In this work, we introduce Wonder3D, a novel method for efficiently generating high-fidelity textured meshes from single-view images. Recent methods based on Score Distillation Sampling (SDS) have shown the potential to recover 3D geometry from 2D diffusion priors, but they typically suffer from time-consuming per-shape optimization and inconsistent geometry. In contrast, certain works directly produce 3D information via fast network inferences, but their results are often of low quality and lack geometric details. To holistically improve the quality, consistency, and efficiency of single-view reconstruction tasks, we propose a cross-domain diffusion model that generates multiview normal maps and the corresponding color images. To ensure the consistency of generation, we employ a multiview cross-domain attention mechanism that facilitates information exchange across views and modalities. Lastly, we introduce a geometry-aware normal fusion algorithm that extracts high-quality surfaces from the multi-view 2D representations in only 2 ∼ 3 minutes. Our extensive evaluations demonstrate that our method achieves high-quality reconstruction results, robust generalization, and good efficiency compared to prior works.  1. Introduction  Reconstructing 3D geometry from a single image [13, 26, 33, 38, 40, 42, 45] stands as a fundamental task in computer graphics and 3D computer vision, benefiting a wide range of versatile applications such as novel view synthesis [7, 25, 35], 3D content creation [36, 48], and robotics grasping [29, 79]. However, this task is notably challenging since it is ill-posed and demands the ability to discern the 3D geometry of both visible and invisible parts. This ability requires extensive knowledge of the 3D world. Recently, the field of 3D generation has experienced rapid and flourishing development with the introduction of diffusion models. A growing body of research [5, 31, 47, 63, 67], such as DreamField [24], DreamFusion [47], and Magic3D [31], resort to distilling prior knowledge of 2D image diffusion models or vision language models to create 3D models from text or images via Score Distillation Sampling (SDS) [47]. Despite their compelling results, these methods suffer from two main limitations: efficiency and consistency. The per-shape optimization process typically entails tens of thousands of iterations, involving full-image volume rendering and inferences of the diffusion models. Consequently, it often consumes tens of minutes or even hours on per-shape optimization. Moreover, the 2D prior model operates by considering only a single view at each iteration and strives to make every view resemble the input image. This often results in the generation of 3D shapes  exhibiting inconsistencies, thus, often leading to the generation of 3D shapes with inconsistencies such as multiple faces (i.e., the Janus problem [47]).  There exists another group of works that endeavor to directly produce 3D geometries like point clouds [41, 45, 75, 80], meshes [16, 37], neural fields [1, 4, 8, 15, 17, 21, 2628, 44, 46, 65, 76] via network inference to avoid timeconsuming per-shape optimization. Most of them attempt to train 3D generative diffusion models from scratch on 3D assets. However, due to the limited size of publicly available 3D datasets, these methods demonstrate poor generalizability, most of which can only generate shapes on specific categories.  More recently, several methods have emerged that directly generate multi-view 2D images, with representative works including SyncDreamer [36] and MVDream [55]. By enhancing the multi-view consistency of image generation, these methods can recover 3D shapes from the generated multi-view images. Our method also adopts a multi-view generation scheme to favor the flexibility and efficiency of 2D representations. However, due to only relying on color images, the fidelity of the generated shapes is not wellmaintained, and they struggle to recover geometric details or come with enormous computational costs.  To better address the issues of fidelity, consistency, generalizability and efficiency in the aforementioned works, in this paper, we introduce a new approach to the task of single-view 3D reconstruction by generating multi-view consistent normal maps and their corresponding color images with a cross-domain diffusion model. The key idea is to extend diffusion frameworks to model the joint distribution of two different domains, i.e., normals and colors. The normal map characterizes the undulations and variations presented on the surface of the shape, thus encoding rich detailed geometric information. This allows for the highfidelity extraction of 3D geometry from 2D normal maps. Adopting the 2D representations enables Wonder3D to be built on the pre-trained Stable Diffusion model [49], where strong priors facilitate zero-shot generalization ability.  The following technical designs of Wonder3D make it a robust and efficient tool to create 3D shapes from single images. 1) Cross-domain switcher. The introduced domain switcher allows the diffusion model to generate either normal maps or color images without significantly modifying the original model. 2) Cross-domain attentions. We further leverage cross-domain attention mechanisms to assist in the information exchange between the two domains, ultimately improving consistency and quality. This mechanism facilitates information perception across different domains, enabling our method to recover high-fidelity geometry. 3) Geometry-aware normal fusion. In order to stably extract surfaces from the generated views, we propose a geometryaware normal fusion algorithm that is robust to subtle inac CLIP  Geometry Fusion  Normal Color  ⨁ ⋯  ⋯  Cameras Domain switcher  Single RGB image Generated normals and colors  Textured mesh  UNet  Figure 2. Overview of Wonder3D. Given a single image, Wonder3D takes the input image, the text embedding produced by CLIP model [49], the camera parameters of multiple views, and a domain switcher as conditioning to generate consistent multi-view normal maps and color images. Subsequently, Wonder3D employs an innovative normal fusion algorithm to robustly reconstruct high-quality 3D geometry from the 2D representations, yielding high-fidelity textured meshes.  curate generations and capable of reconstructing clean and high-quality geometries (see Figure 1). We conduct extensive experiments on the Google Scanned Object dataset [14] and various 2D images with different styles. The experiments validate that Wonder3D achieves a leading level of geometric details with high efficiency and strong generalization among current zero-shot single-view reconstruction methods.  2. Related Works  2.1. 2D Diffusion Models for 3D Generation  Recent compelling successes in 2D diffusion models [9, 22, 51] and large vision language models (e.g., CLIP model [49]) provide new possibilities for generating 3D assets using the strong priors of 2D diffusion models. Pioneering works DreamFusion [47] and SJC [63] propose to distill a 2D text-to-image generation model to generate 3D shapes from texts, and many follow-up works follow such per-shape optimization scheme. For the task of textto-3D [2, 5, 6, 23, 31, 52, 53, 61, 67, 69, 73, 82] or imageto-3D synthesis [42, 48, 50, 54, 58, 71], these methods typically optimize a 3D representation (i.e., NeRF, mesh, or SDF), and then leverage neural rendering to generate 2D images from various viewpoints. The images are then fed into the 2D diffusion models or CLIP model for calculating SDS [47] losses, which guide the 3D shape optimization. However, most of these methods always suffer from low efficiency and multi-face problem, where a per-shape optimization consumes tens of minutes and the optimized geometry tends to produce multiple faces due to the lack of explicit 3D supervision.  2.2. 3D Generative Models  Instead of performing a time-consuming per-shape optimization guided by 2D diffusion models, some works attempt to directly train 3D diffusion models based on various 3D representations, like point clouds [41, 45, 75, 80],  meshes [16, 37], neural fields [1, 4, 8, 15, 17, 21, 2628, 44, 46, 65, 76] However, due to the limited size of public available 3D assets dataset, most of the works have only been validated on limited categories of shapes, and how to scale up on large datasets is still an open problem. On the contrary, our method adopts 2D representations and, thus, can be built upon the 2D diffusion models [51] whose pretrained priors significantly facilitate zero-shot generalization ability.  2.3. Multi-view Diffusion Models  To generate consistent multi-view images, some efforts [3, 11, 18, 30, 34, 57, 59, 60, 62, 68, 70, 72, 74, 77, 81] are made to extend 2D diffusion models from single-view images to multi-view images. However, most of these methods focus on image generation and are not designed for 3D reconstruction. The recent works Viewset Diffusion [57], SyncDreamer [36], and MVDream [55] share a similar idea to produce consistent multi-view color images via attention layers. However, unlike that normal maps explicitly encode geometric information, reconstruction from color images always suffers from texture ambiguity, and, thus, they either struggle to recover geometric details or require huge computational costs. SyncDreamer [36] requires dense views for 3D reconstruction but still suffers from low-quality geometry and blurring textures. MVDream [55] still resorts to a time-consuming optimization using SDS loss for 3D reconstruction, and its multi-view distillation scheme requires 1.5 hours. In contrast, our method can reconstruct high-quality textured meshes in just 2 minutes
GSD
GSD: View-Guided Gaussian Splatting Diffusion
for 3D Reconstruction
Yuxuan Mu
1
,
2
⋆
, Xinxin Zuo
2
, Chuan Guo
1
,
2
, Yilin Wang
1
,
2
, Juwei Lu
2
,
Xiaofeng Wu
2
, Songcen Xu
2
, Peng Dai
2
, Youliang Yan
2
, and Li Cheng
1
1 University of Alberta, Edmonton AB T6G 2R3, Canada
{ymu3, <PRESIDIO_ANONYMIZED_EMAIL_ADDRESS> 2 Huawei Noah’s Ark Lab
Abstract. We present GSD, a diffusion model approach based on Gaussian Splatting (GS) representation for 3D object reconstruction from a
single view. Prior works suffer from inconsistent 3D geometry or mediocre
rendering quality due to improper representations. We take a step towards resolving these shortcomings by utilizing the recent state-of-theart 3D explicit representation, Gaussian Splatting, and an unconditional
diffusion model. This model learns to generate 3D objects represented
by sets of GS ellipsoids. With these strong generative 3D priors, though
learning unconditionally, the diffusion model is ready for view-guided
reconstruction without further model fine-tuning. This is achieved by
propagating fine-grained 2D features through the efficient yet flexible
splatting function and the guided denoising sampling process. In addition, a 2D diffusion model is further employed to enhance rendering fidelity, and improve reconstructed GS quality by polishing and reusing the rendered images. The final reconstructed objects explicitly
come with high-quality 3D structure and texture, and can be efficiently
rendered in arbitrary views. Experiments on the challenging real-world
CO3D dataset demonstrate the superiority of our approach. Project
page: https://yxmu.foo/GSD/
Keywords: Object Reconstruction, Gaussian Splatting, Guided Diffusion Model
1 Introduction
Given the abundance of image data in the real world, the problem of 3D reconstruction from single-view images has garnered notable attention [11, 41,44, 49].
While humans can effortlessly deduce the general object shape and even imagine
its texture from unseen views, for computational models, the problem becomes
highly non-trivial. As will be described next, three key aspects underpin this
problem. First, a proper 3D representation capable of encoding high-fidelity 3D
information, while being compatible with various levels of quantization. Second,
akin to the human perception system, it is crucial to have a generative model ⋆ This work was done during an internship at Huawei Noah’s Ark Lab
arXiv:2407.04237v3 [cs.CV] 19 Jul 2024
2 Y. Mu et al. GS ~ N(0, I) GS Denoising GS x0
View
-Guided Sampling
Fig. 1: A illustration of our View-Guided Gaussian Splatting Diffusion framework for single-view 3D reconstruction. It works by progressively denoising a randomly
initialized set of Gaussian Splatting (GS) ellipsoids with continuous guidance from the
discrepancies between the input and rendered images. The gray arrow represents the
splatting-based GS rendering, while the orange arrow depicts the backpropagation of
guidance gradients. The diffusion model built directly upon the GS representation in
our context provides explicit geometry information. The view-guided sampling takes
the advantage of splatting function to faithfully yet efficiently obtain fine-grained features from the given view.
being able to produce an object with diverse appearances of the object’s backside and being faithful to the input views. Finally, the ability to efficiently and
precisely render a 3D object into an arbitrary view.
Existing efforts often struggle to properly address one or multiple aspect(s)
of the above three. For instance, the recent 2D novel view synthesis methods [3,18,21,31,42,52] usually fall short in maintaining 3D consistency. Another
line of research [11, 12, 19, 20, 24, 41, 44, 45, 47] based on explicit 3D representations such as voxels, point clouds and meshes, are limited to coarse geometry and
suboptimal rendering quality, though offering consistent 3D rendering. This may
be attributed to the low resolution or sparse nature of 3D features (e.g., points,
voxels) and the challenges of their discretization (e.g., mesh) in deep learning
models. Implicit 3D representation, on the other hand, formulates 3D space as
a query-based implicit function [1, 7, 25, 26, 33], achieving remarkable quality in
single-view 3D reconstruction in well-defined canonical space [5, 15, 49]. Unfortunately, they typically rely on cumbersome efforts such as marching cubes for
3D geometry extraction and view rendering.
Motivated by these observations, we introduce a novel framework, GSD, for
high-quality single-view 3D reconstruction by building a generative Diffusion
Transformer (DiT) [28] upon the emerging Gaussian Splatting (GS) representation [17]. Specifically, GS encodes a scene by a set of GS ellipsoids, with each
ellipsoid parameterized by its center position, covariance, regional color, and
opacity. Unlike existing 3D representations, GS explicitly encodes 3D geome-
View-Guided GS Diffusion for 3D Reconstruction 3
Fig. 2: Unconditional Generation of GS DiT on the Hydrant dataset. Ten
distinct samples are generated from our unconditional diffusion model which is trained
on more than 500 hydrant scenes using the GS representation. Our diffusion model
shows an appealing ability on modeling the generative priors of 3D objects.
try and texture in high resolution and density. Furthermore, due to its spatial
explicitness, we can easily deploy a point-space DiT even without positional
encoding [27]. Different from other diffusion models using classifier-free conditioning with image encoders [23,27,52], our GS DiT enables efficient fine-grained
image conditioning through its unique splatting-based rendering and loss-guided
sampling [35], which also ensures fidelity to the given-view images.
Upon the GS representation, a category-specific 3D DiT is trained to capture the space of plausible 3D objects in terms of their diverse geometries and
textures. As illustrated in Fig. 2, when w/o an input image, our diffusion model
learns to generate high-fidelity 3D objects with distinct geometries and textures.
When an input image is provided, the same diffusion model is used to reconstruct
the specific 3D object, faithful when rendering to the same view. This process
is presented in Fig. 1. During test time, the GS object at each denoising step is
projected to the given view through the differentiable splatting-based rendering.
The gradients of discrepancies between the rendered and the reference images are
then backpropagated to the corresponding GS samples to refine the 3D object at
the current step, similar to classifier guidance [35]. This approach is simple yet
effective, and can be easily adapted to multi-view reconstruction. In addition,
an auxiliary 2D diffusion model is employed to further improve the quality of
rendered images, which reciprocally facilitates better 3D reconstruction.
Our main contributions can be summarized as follows:
– Our proposed GSD is, to our knowledge, the first diffusion model that directly models raw GS representation capturing its 3D generative prior for
single-view reconstruction.
– The GS DiT intuitively comes with an effective yet flexible view-guided sampling strategy that can extract fine-grained features from the given views
using the efficient splatting function. Given an input image at test-time,
the guided iterative denoising of our GS-based diffusion model allows a progressive refinement of the reconstructed 3D object consistent with the input
view.
4 Y. Mu et al.
– Empirical experiments of the real-world CO3D dataset demonstrate the superiority of our approach when compared to the state-of-the-art. Our approach is flexible and can also work with multi-view images.
2 Related Work
View-Conditioned 3D Reconstruction and Generation. Many related
works reconstruct the 3D shape by jointly modeling the unconditional 3D priors
and the conditional distribution with a generative model [16, 27, 50]. Some of
them working on explicit 3D representations [24, 27, 44, 50] can recover explicit
geometry but fail to synthesize photorealistic views. For the other stream of
studies [1, 3, 5, 16, 26, 33], the use of advanced implicit representations enables
photo-quality view synthesis while struggling to extract accurate geometry in
unconstrained space. We find the emerging 3D representation Gaussian Splatting [17] has great potential to be a generally suitable representation for this task,
which enjoys both benefits from explicit geometry and realistic view synthesis.
While concurrent works using GS representation either re-form a deterministic prediction problem [37] or combine with latent representation for additional
feature decoration [4, 46, 53]. Moreover, most of the previous works perceive
the given view in the camera space by an image encoder [16, 27, 38], which
cannot ensure faithful reconstruction due to the compression and also requires
canonical coordinates to constrain the modeling space. They indeed should be
image-conditioned generation, rather than reconstruction from views. Inspired
by the fine-grained projection method from PC2
[24], we take advantage of GS
splatting-based rendering [17] to get access to the image through pixel-level gradients that reliably keep the view information and are flexible to accommodate
arbitrary views using relative camera parameters in world space. A similar gradient conditioning approach is also used in SSD NeRF [5], while restricted by
the robustness of its dataset-specific neural rendering. Combining GS universal
rendering with view-guided sampling conditioning on the GS diffusion model
could fully explore the potential of these methods.
Novel View Synthesis. The current novel view synthesis (NVS) task becomes
progressively close to the 3D reconstruction task [18,31,32,52]. One of the most
significant pinpoints is the 3D inconsistency issue from being short of 3D geometry priors. To address this problem, some works try to involve multi-view geometry [49,52], depth [2,23], and clues from large multi-view 2D dataset [21,23,32,42].
However, since it primarily focuses on imaging geometry awareness rather than
modeling the prior distribution of 3D shapes. We argue this 2D objective deviates
from 3D reconstruction, potentially resulting in unsatisfactory 3D geometry.
SDS-based 3D Asset Creation. 3D asset creation emerges thanks to the
boom of big models. Most of them distill the 3D representation from a pretrained image generation model by Score Distillation Sampling (SDS) [6,29,39].
One of the primary weaknesses lies in 3D geometry, despite involving multiview geometry to regress the 3D representation. These approaches heavily rely
on the consistent performance of large pre-trained image models, where views
View-Guided GS Diffusion for 3D Reconstruction 5
are treated as independent during pre-training. This could lead to the Janus
problem. Another issue arises when applying these methods to our object reconstruction scenario, as the 3D space should ideally be well-constrained. However,
achieving alignment in a real-world setting is challenging. Consequently, the 3D
asset creation approaches may not easily adapt to practical applications in realworld object reconstruction.