以下是输入草图和文本生成3d的几篇工作，假设你是一位对这个领域了解非常清楚的资深专家，请问能否对这些工作的相似点和不同点做一个总结，分析一下可能存在的研究空白？
MVcontrol
We introduce MVControl, a novel neural network architecture that enhances existing pre-trained multi-view 2D diffusion models by incorporating additional input conditions, e.g. edge maps. Our approach enables the generation of controllable multi-view images and view-consistent 3D content. To achieve controllable multi-view image generation, we leverage MVDream as our base model, and train a new neural network module as additional plugin for endto-end task-specific condition learning. To precisely control the shapes and views of generated images, we innovatively propose a new conditioning mechanism that predicts an embedding encapsulating the input spatial and view conditions, which is then injected to the network globally. Once MVControl is trained, score-distillation (SDS) loss based optimization can be performed to generate 3D content, in which process we propose to use a hybrid diffusion prior. The hybrid prior relies on a pre-trained Stable-Diffusion network and our trained MVControl for additional guidance. Extensive experiments demonstrate that our method achieves robust generalization and enables the controllable generation of high-quality 3D content. Code available at https://github.com/WU-CVGL/MVControl.  1. Introduction  Remarkable progress has been made in the field of 2D image generation recently. High fidelity images can be easily generated via input text prompts [52]. Due to the scarcity of 3D training data, the success in text-to-image generation is hardly copied to the text-to-3D domain. Instead of training a large text-to-3D generative model from scratch with large amounts of 3D data, due to the nice properties of diffusion models [21, 61] and differentiable 3D representations [27, 40, 57, 71], recent score distillation optimization (SDS) [48] based methods [12, 31, 39, 48, 65, 66, 73], attempt to distill 3D knowledge from a pre-trained large text† Corresponding author.  to-image generative model [52] and have achieved impressive results. The representative work is DreamFusion [48], which starts a new paradigm for 3D asset generation. Following the 2D-to-3D distillation methodology, the techniques are rapidly evolving over the past year. Many works have been proposed to further improve the generation quality by applying multiple optimization stages [12, 31], optimizing the diffusion prior with the 3D representation simultaneously [62, 73], deriving more precise formulation of score distillation algorithm [26, 83], or enhancing the details of whole pipeline [4, 23, 87]. Although the above mentioned efforts can get access to delicate texture, the view consistency of generated 3D content is hard to be achieved, since the 2D diffusion prior is not viewdependent. Hence, there is a series of works hammering at introducing multi-view knowledge to the pre-trained diffusion models [30, 33–35, 58, 59]. Although they can deliver impressive text controlled multiview images and 3D assets, they still cannot achieve fine-grained control over the generated content via an edge map for example, as its counterpart in text-to-image generation, i.e. ControlNet [85]. In this work, we therefore propose MVControl, a multi-view version of ControlNet, to enable controllable text-to-multiview image generation. Once MVControl is trained, we can exploit it to the score distillation optimization pipeline, so as to achieve controllable text-to-3D content generation via an input condition image, e.g. edge map. Inspired by 2D ControlNet [85], which works as a plugin module of Stable-Diffusion [52], we choose a recently released multi-view diffusion network, MVDream [59], as our base model. A control network is then designed to interact with the base model to achieve controllable text-tomulti-view image generation. Similarly to [85], the weights of MVDream is all frozen and we only train the control network. While MVDream is trained with camera poses defined in the absolute world coordinate system, we experimentally find that the relative pose condition with respect to the condition image is more proper for controllable text-tomulti-view generation. However, it conflicts with the definition of the pretrained MVDream network. Furthermore,  arXiv:2311.14494v2 [cs.CV] 27 Nov 2023 Figure 1. MVControl: Given an input text prompt and an edge map, our method is able to generate high-fidelity controllable multi-view images and view-consistent 3D content.  since the conditioning mechanism of 2D ControlNet is designed for single image generation and does not consider the multi-view scenario, view-consistency cannot be easily achieved by directly adopting its control network to interact with the base model. To overcome these issues, we design a simple but effective novel conditioning strategy based on the original ControlNet architecture to achieve controllable text-to-multi-view generation. MVControl is jointly trained on a subset of the large-scale 2D dataset LAION [55] and 3D dataset Objaverse [16] as what [59] does. We only explore to use the edge map as conditional input in this work. However, our network has no restriction to use other type of input conditions, e.g. depth map, sketch image etc. Once MVControl is trained, we can exploit it to provide 3D priors for controllable text-to-3D asset generation. In particular, we employ a hybrid diffusion prior relying on pretrained Stable-Diffusion model and MVControl network. The generation is conducted in a coarse-to-fine manner. After we get a good geometry after the coarse stage, we fix it and only optimize the texture during the fine stage. Our extensive experiments demonstrate that our proposed method can generate fine-grain controlled high-fidelity multi-view images and 3D content via an input condition image as well as textual description. In summary, our main contributions are as follows.  • We propose a novel network architecture to achieve fine-grain controlled text-to-multi-view image generation;  • Once our network is trained, it can be exploited to serve as a part of hybrid diffusion prior for controllable text-to-3D content generation vis SDS optimization;  • Extensive experimental results demonstrate that our method is able to deliver high-fidelity multi-view image and 3D asset, which can be fine-grain controlled by an input condition image and text prompt;  • Besides being used to generate 3D asset via SDS optimization, we believe our MVControl network could benefit the general 3D vision/graphic community for broad application scenarios.

Sketchdream
Existing text-based 3D generation methods generate attractive results but lack detailed geometry control. Sketches, known for their conciseness and expressiveness, have contributed to intuitive 3D modeling but are confined to producing texture-less mesh models within predefined categories. Integrating sketch and text simultaneously for 3D generation promises enhanced control over geometry and appearance but faces challenges from 2D-to-3D translation ambiguity and multi-modal condition integration. Moreover, further editing of 3D models in arbitrary views will give users more freedom to customize their models. However, it is difficult to achieve high generation quality, preserve unedited regions, and manage proper interactions between shape components. To solve the above issues, we propose a text-driven 3D content generation and editing method, SketchDream, which supports NeRF generation from given hand-drawn sketches and achieves free-view sketchbased local editing. To tackle the 2D-to-3D ambiguity challenge, we introduce a sketch-based multi-view image generation diffusion model, which leverages depth guidance to establish spatial correspondence. A 3D ControlNet with a 3D attention module is utilized to control multi-view images and ensure their 3D consistency. To support local editing, we further propose a coarse-to-fine editing approach: the coarse phase analyzes component  ∗Corresponding author: Lin Gao (<PRESIDIO_ANONYMIZED_EMAIL_ADDRESS>).  Authors’ addresses: Lin Gao and Feng-Lin Liu are with the Beijing Key Laboratory of Mobile Computing and Pervasive Device, Institute of Computing Technology, Chinese Academy of Sciences, and University of Chinese Academy of Sciences. Hongbo Fu was with the SCM, City University of Hong Kong, and is now with EMIA, HKUST. YuKun Lai is with the School of Computer Science and Informatics at Cardiff University. Authors’ e-mails: <PRESIDIO_ANONYMIZED_EMAIL_ADDRESS>, <PRESIDIO_ANONYMIZED_EMAIL_ADDRESS>, <PRESIDIO_ANONYMIZED_EMAIL_ADDRESS>, <PRESIDIO_ANONYMIZED_EMAIL_ADDRESS>. This is the author’s version of the work. It is posted here for your personal use. Not for redistribution.  interactions and provides 3D masks to label edited regions, while the fine stage generates realistic results with refined details by local enhancement. Extensive experiments validate that our method generates higher-quality results compared with a combination of 2D ControlNet and image-to-3D generation techniques and achieves detailed control compared with existing diffusion-based 3D editing approaches.  CCS Concepts: • Human-centered computing → Graphical user interfaces; • Computer systems organization → Neural networks; • Computing methodologies → Rendering; Volumetric models.  Additional Key Words and Phrases: sketch-based interaction, diffusion models, neural radiance fields, 3D generation  1 INTRODUCTION  Creating high-quality 3D content is a popular topic with wide applicability in VR/AR, the movie industry, architecture, robotics simulation, etc. However, traditional 3D content production depends on sophisticated software and laborious procedures, making it challenging for amateur users to design their 3D models. To solve this problem, sketching, a user-friendly and expressive interaction tool, has been utilized in 2D [Chen et al. 2009; Isola et al. 2017; Zhang et al. 2023] and 3D content generation [Gao et al. 2022, 2023b; Xiang et al. 2020; Zhang et al. 2021a; Zheng et al. 2023]. However, due to the restriction of network capacity and the ambiguity of 2D sketches, existing works can only generate limited single or multiple categories of objects. Additionally, since sketches only contain geometry  arXiv:2405.06461v2 [cs.GR] 14 May 2024 information, how to effectively control the appearance has been under-explored in existing sketch-based 3D generation works. Compared with sketches, text prompts can describe object category and appearance more easily. Thanks to the development of diffusion models, text-to-image generation [Midjournal. 2022; Rombach et al. 2022] has become successful in recent years. Based on these pre-trained 2D models, DreamFusion [Poole et al. 2023] designs Score Distillation Sampling (SDS) to optimize Neural Radiance Fields (NeRFs) and presents a text-to-3D generation framework. Follow-up works [Chen et al. 2023a; Lin et al. 2023; Qiu et al. 2023; Shi et al. 2023b; Wang et al. 2023c] further modify the supervision losses and optimization process to generate more realistic and higher-quality 3D models. Despite impressive results of text-based 3D generation, text prompts cannot precisely depict objects’ shape, texture patterns, and layouts, and are thus less suitable for finegrained control. Moreover, it is difficult for the above works to edit the local details of generated or real 3D models because of their global control of textual description. Although some works [Cheng et al. 2023; Sella et al. 2023; Zhuang et al. 2023] further achieve text-based local editing by labeling editing regions, it is still hard for them to easily control the shape and position of edited components. To improve the controllability of 3D generation, we introduce sketch into the text-to-3D generation framework: sketch guides the shape and pattern, while text prompt controls the material and appearance. This goal is challenging to achieve for the following reasons. First, since a sketch only contains single-view information, the absence of multi-view supervision makes it difficult to generate complete 3D models. Moreover, directly adding single-view sketch constraints in other viewpoints degrades the generation quality and has lower faithfulness. For example, a straightforward approach [Zeng et al. 2023] is to first generate 2D images by ControlNet [Zhang et al. 2023] with a sketch-based condition and then utilize image-to-3D approaches [Liu et al. 2023b; Qian et al. 2023; Sun et al. 2023] to generate 3D contents. However, as shown in Fig. 8, this approach tends to generate distorted geometry in novel views different from input sketches and fuzzy texture details in the back view due to the information absence. Further editing of the generated contents in other views provides users with more detailed control of results. Additionally, editing existing real models instead of generation from scratch is also a common situation. However, sketch-based 3D content editing is challenging because besides achieving high-quality content creation, the edited content should reasonably interact with the original content, with unedited regions well preserved. To solve the above issue, SKED [Mikaeili et al. 2023] requires the input of multi-view sketches, which are treated as binary masks to label edited regions. Although it achieves effective text- and sketch-based editing at the object level, maintaining unedited regions well and supporting part-level editing are still challenging. To address the above issues, we propose SketchDream, a method for sketch-based text-to-3D generation and editing of photo-realistic contents. For sketch-based generation, since sketches are sparse and lack 3D information, it is ambiguous and difficult to directly generate 3D contents from them. To solve this problem, we use the depth information, which can bridge the 2D inputs and 3D models. Specifically, given an input 2D sketch and a text prompt,  we utilize a 2D diffusion model [Zhang et al. 2023] to generate a corresponding depth map, which is then utilized to warp the input sketch. To propagate the sketch into 3D space and avoid the Janus problem (i.e., 3D models with multiple frontal faces), we build a 3D ControlNet based on MVDream [Shi et al. 2023b] to generate images in four camera views uniformly distributed in azimuth. To build the correspondence between the sketch and multi-view images, we modify the MVDream backbone to generate an additional image in the view of the input sketch. To ensure the consistency between different views and support sketch control, we design a 3D-attention control module, which takes the input 2D sketch and warped sketch in the nearby view with depth guidance as inputs to effectively control the 3D diffusion generation. Similar to MVDream [Shi et al. 2023b], we use SDS-based optimization to generate high-quality 3D contents. Our framework naturally supports sketch-based 3D content editing. Given the generated or real 3D models, users can input an edited sketch with a 2D editing mask to modify local components. To generate high-quality editing results, we design a two-stage coarse-to-fine editing framework. In the coarse stage, the 2D mask is lifted into 3D space to construct a coarse 3D columnar mask. Then, the coarse stage generates an editing result that generally conforms to the edited sketch and the input text, but might have unoptimized quality, inadequate faithfulness, and mistaken interaction with the original object. We optimize the editing result quality in the fine stage. In particular, we extract a mesh model from the coarse-stage editing result and then label the edited region by the coarse 3D columnar mask. This new 3D mask precisely represents the component interaction and helps preserve the unedited regions. Moreover, we propose a local rendering strategy that calculates the sketchbased SDS supervision locally to enhance the sketch faithfulness and generation quality further. Extensive experiments validate that our method generates higherquality results than possible sketch-based text-to-3D baselines and existing sketch-based 3D editing approaches. Our main contributions can be summarized as follows.  • We propose the first sketch-based text-to-3D generation and editing method, which generates high-quality 3D objects under generalized categories and supports detailed editing of reconstructed or generated NeRFs. • We propose a sketch-based multi-view image generation diffusion model, which utilizes a depth-guided warping strategy to create spatial correspondence, and a 3D-attention control module to ensure 3D consistency. • To support local modification further, we develop a coarseto-fine editing framework. The coarse stage generates initial results to label edited regions better, while the fine stage generates high-quality editing results with a local rendering strategy.

Sketch3d
Recently, image-to-3D approaches have achieved significant results with a natural image as input. However, it is not always possible to access these enriched color input samples in practical applications, where only sketches are available. Existing sketch-to-3D researches suffer from limitations in broad applications due to the challenges of lacking color information and multi-view content. To overcome them, this paper proposes a novel generation paradigm Sketch3D to generate realistic 3D assets with shape aligned with the input sketch and color matching the textual description. Concretely, Sketch3D first instantiates the given sketch in the reference image through the shape-preserving generation process. Second, the reference image is leveraged to deduce a coarse 3D Gaussian prior, and multi-view style-consistent guidance images are generated based on the renderings of the 3D Gaussians. Finally, three strategies are designed to optimize 3D Gaussians, i.e., structural optimization via a distribution transfer mechanism, color optimization with a straightforward MSE loss and sketch similarity optimization with a CLIP-based geometric similarity loss. Extensive visual comparisons and quantitative analysis illustrate the advantage of our Sketch3D in generating realistic 3D assets while preserving consistency with the input.

Sketch-a-shape
Significant progress has recently been made in creative applications of large pre-trained models for downstream tasks in 3D vision, such as text-to-shape generation. This motivates our investigation of how these pre-trained models can be used effectively to generate 3D shapes from sketches, which has largely remained an open challenge due to the limited sketch-shape paired datasets and the varying level of abstraction in the sketches. We discover that conditioning a 3D generative model on the features (obtained from a frozen large pre-trained vision model) of synthetic renderings during training enables us to effectively generate 3D shapes from sketches at inference time. This suggests that the large pre-trained vision model features carry semantic signals that are resilient to domain shifts, i.e., allowing us to use only RGB renderings, but generalizing to sketches at inference time. We conduct a comprehensive set of experiments investigating different design factors and demonstrate the effectiveness of our straightforward approach for generation of multiple 3D shapes per each input sketch regardless of their level of abstraction without requiring any paired datasets during training.

Control3d
Recent remarkable advances in large-scale text-to-image diffusion models have inspired a significant breakthrough in text-to-3D generation, pursuing 3D content creation solely from a given text prompt. However, existing text-to-3D techniques lack a crucial ability in the creative process: interactively control and shape the synthetic 3D contents according to users' desired specifications (e.g., sketch). To alleviate this issue, we present the first attempt for text-to-3D generation conditioning on the additional hand-drawn sketch, namely Control3D, which enhances controllability for users. In particular, a 2D conditioned diffusion model (ControlNet) is remoulded to guide the learning of 3D scene parameterized as NeRF, encouraging each view of 3D scene aligned with the given text prompt and hand-drawn sketch. Moreover, we exploit a pre-trained differentiable photo-to-sketch model to directly estimate the sketch of the rendered image over synthetic 3D scene. Such estimated sketch along with each sampled view is further enforced to be geometrically consistent with the given sketch, pursuing better controllable text-to-3D generation. Through extensive experiments, we demonstrate that our proposal can generate accurate and faithful 3D scenes that align closely with the input text prompts and sketches.