Sketch3D: Style-Consistent Guidance for Sketch-to-3D Generation

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-30 15:26:02

Abstract: Recently, image-to-3D approaches have achieved significant results with a natural image as input. However, it is not always possible to access these enriched color input samples in practical applications, where only sketches are available. Existing sketch-to-3D researches suffer from limitations in broad applications due to the challenges of lacking color information and multi-view content. To overcome them, this paper proposes a novel generation paradigm Sketch3D to generate realistic 3D assets with shape aligned with the input sketch and color matching the textual description. Concretely, Sketch3D first instantiates the given sketch in the reference image through the shape-preserving generation process. Second, the reference image is leveraged to deduce a coarse 3D Gaussian prior, and multi-view style-consistent guidance images are generated based on the renderings of the 3D Gaussians. Finally, three strategies are designed to optimize 3D Gaussians, i.e., structural optimization via a distribution transfer mechanism, color optimization with a straightforward MSE loss and sketch similarity optimization with a CLIP-based geometric similarity loss. Extensive visual comparisons and quantitative analysis illustrate the advantage of our Sketch3D in generating realistic 3D assets while preserving consistency with the input.

Main paper: 
 related work 2.1 text-to-3d generation: 
 Text-to-3D generation aims at generating 3D assets from a text prompt. Recent developments in text-to-image methods [37] [38] [39] have demonstrated a remarkable capability to generate high-quality and creative images from given text prompts. Transferring it to 3D generation presents non-trivial challenges, primarily due to the difficulty in curating extensive and diverse 3D datasets. Existing 3D diffusion models [7, 9, 12, 24, 29, 31, 55, 60] typically focus on a limited number of object categories and face challenges in generating realistic 3D assets. To accomplish generalizable 3D generation, innovative works like DreamFusion [32] and SJC [48] utilize pre-trained 2D diffusion models for text-to-3D generation and demonstrate impressive results. Following works continue to enhance various aspects such as generation fidelity and efficiency [4, 11, 18, 20, 44, 46, 49, 54, 62] , and explore further applications [1, 36, 42, 63] . However, the generated contents of text-to-3D method are unpredictable and the shape cannot be controlled according to user requirements.Sketch-to-3D generation aims to generate 3D assets from a sketch image and possible text input. Since sketches are highly abstract and lack substantial information [41] , generating 3D assets based on sketches becomes a challenging problem. Sketch2Model [58] introduces an architecture for view-aware generation that explicitly conditions the generation process on specific viewpoints. Sketch2Mesh [8] employs an encoder-decoder architecture to represent and adjust a 3D shape so that it aligns with the target external contour using a differentiable renderer. SketchSampler [6] proposes a sketch translator module to utilize the spatial information within a sketch and generate a 3D point cloud that represents the shape of the sketch. Sketch-A-Shape [40] proposes a zero-shot approach for sketch-to-3D generation, leveraging large-scale pre-trained models.SketchFaceNeRF [19] proposes a sketch-based 3D facial NeRF generation and editing method. SKED [26] proposes a sketch-guided 3D editing technique to edit a NeRF. Overall, existing sketch-to-3D generation methods have several limitations. First, generating 3D shapes from sketches invariably produces shapes without color information and needs to be trained on large-scale datasets, yet the trained models are typically limited to making predictions on a single category. Second, the 3D assets generated by sketch-guided generation or editing techniques often lack realism, and the process is relatively time-consuming. Our method, incorporating the input text prompt, is capable of generating 3D assets with shapes consistent with the sketch and color aligned with the textual description. 
method: 
 In this section, we first introduce two preliminaries including 3D Gaussian Splatting and Controllable Image Synthesis (Sec. 3.1). Subsequently, we systematically propose our Sketch3D framework (Sec. 3.2), which is progressively introduced (Sec. 3.3-3.5).3D Gaussian Splatting. 3D Gaussian Splatting (3DGS) [13] represents a novel method for novel-view synthesis and 3D scene reconstruction, achieving promising results in both quality and real-time processing speed. Unlike implicit representation methods such as NeRF [27] , 3D Gaussians represent the scene through a set of anisotropic Gaussians, defined with its center position 𝜇 ∈ R 3 , covariance Σ ∈ R 7 , color c ∈ R 3 , and opacity 𝛼 ∈ R 1 . The covariance matrix Σ = RSS ⊤ R ⊤ describes the configuration of an ellipsoid and is implemented via a scaling matrix S and a rotation matrix R.Each Gaussian centered at point (mean) 𝜇 is defined as:EQUATIONwhere 𝑥 represents the distance between 𝜇 and the query point.A ray 𝑟 is cast from the center of the camera, and the color and density of the 3D Gaussians that the ray intersects are computed along the ray. In summary, 𝐺 (𝑥) is multiplied by 𝛼 in the blending process to construct the final accumulated color:EQUATIONwhere 𝑁 means the number of samples on the ray 𝑟 , 𝑐 𝑖 and 𝛼 𝑖 denote the color and opacity of the 𝑖-th Gaussian. Controllable Image Synthesis. In the field of image generation, achieving control over the output remains a great challenge. Recent efforts [15, 17, 59 ] have focused on increasing the controllability of generated images by various methods. This involves increasing the ability to specify various attributes of the generated images, such as shape and style. ControlNet [57] and T2I-adapter [28] attempt to control image creation utilizing data from different visual modalities. Specifically, ControlNet is an end-to-end neural network architecture that controls a diffusion model (Stable Diffusion [38] ) to adapt task-specific input conditions. IP-Adapter [51] and MasaCtrl [2] leverage the attention layer to incorporate information from additional images, thus achieving enhanced controllability over the generated results.Given a sketch image and a corresponding text prompt, our objective is to generate realistic 3D assets that align with the shape of the sketch and correspond to the color described in the textual description. To achieve this, we confront three challenges:• How to solve the problem of missing information in sketches?• How to initialize a valid 3D prior from an image? • How to optimize 3D Gaussians to be consistent with the given sketch and the text prompt?Inspired by this motivation, we introduce a novel 3D generation paradigm, named Sketch3D, comprising three dedicated steps to tackle each challenge (as illustrated in Figure 2 ):• Step 1: Generate a reference image based on the input sketch and text prompt (Sec. 3.3). • Step 2: Derive a coarse 3D prior using 3D Gaussian Splatting from the reference image (Sec. 3.4). • Step 3: Generate multi-view style-consistent guidance images through IP-Adapter, introducing three strategies to facilitate the optimization process (Sec. 3.5).For image-to-3D generation, sketches offer very limited information, when served as a visual prompt compared with RGB images. They lack color, depth, semantic information, etc., and only contain simple contours. To solve the above problems, our solution is to create a shapepreserving reference image from a sketch 𝐼 s and a text prompt y. The reference image adheres to the outline of the sketch, while also conforming to the textual description. To achieve this, we leverage an additional image conditioned diffusion model 𝐺 2𝐷 (e.g., ControlNet [57] ) to initiate sketch-preserving image synthesis [5] . Given time step 𝑡, a text prompt y, and a sketch image 𝐼 s , 𝐺 2𝐷 learn a network ε𝜃 to predict the noise added to the noisy image 𝑥 𝑡 with:EQUATIONwhere L is the overall learning objective of 𝐺 2𝐷 . Note that there are two conditions, i.e., sketch 𝐼 s and text prompt y, and the noise is estimated as follows: ε𝜃 (𝑥 𝑡 ; 𝑡, y, 𝐼 s ) =𝜖 𝜃 (𝑥 𝑡 ; 𝑡)EQUATIONwhere 𝑤 is the scale of classifier-free guidance [30] . In summary, 𝐺 2𝐷 can quickly generate a shape-preserving colorful image 𝐼 ref that not only follows the sketch outline but also respects the textual description, which facilitates the subsequent initialization process.A coarse 3D prior can efficiently offer a solid initial basis for subsequent optimization. To facilitate image-to-3D generation, most existing methods rely on implicit 3D representations such as Neural Radiance Fields (NeRF) [45] or explicit 3D representations such as mesh [34] . However, NeRF representations are time-consuming and require high computational resources, while mesh representations have complex representational elements. Consequently, 3D Gaussian Initialization ...Reference Image Distribution Towards (shape colour opacity... ) Gaussian representation, being simple and fast, is chosen as our initialized 3D prior 𝑀 0 .Gaussian Initialization with 3D Diffusion Model. 3D Gaussians can be easily converted from a point cloud, so a simple idea is to first obtain an initial point cloud and then convert it to 3D Gaussians [52] . Therefore, it can be transformed into an image-to-point cloud problem. Currently, many 3D diffusion models use text to generate 3D point clouds [29] . However, we initialize 3D Gaussians 𝑀 0 from 3D diffusion model 𝐺 3𝐷 (e.g., Shap-E [12] ) based on the image 𝐼 ref .Gaussian Initialization through SDS loss. Alternatively, we can also initialize a Gaussian sphere and optimize it into a coarse Gaussian representation through SDS loss [44] . First, we initialize the 3D Gaussians with random positions sampled inside a sphere, with unit scaling and no rotation. At each step, we sample a random camera pose 𝑝 orbiting the object center and render the RGB image 𝐼 𝑝 RGB of the current view. Stable-zero123 [43] is adopted as the 2D diffusion prior 𝜙 and the images 𝐼 𝑝 RGB are given as input. The SDS loss is formulated as:EQUATIONwhere 𝜖 𝜙 (•) is the predicted noise by the 2D diffusion prior 𝜙, and Δ𝑝 is the relative camera pose change from the reference camera. Finally, we can obtain a coarse Gaussian representation 𝑀 0 based on the optimization of the 2D diffusion prior 𝜙.The coarse 3D prior 𝑀 0 is roughly similar in shape to the input sketch, and its color is not completely consistent with the text description. Specifically, the geometric shape generated in Sec. 3.4 may not exactly fit the outline shape of the input sketch 𝐼 s , and there is a certain deviation. For example, the input sketch is an upright, cylinder-like, symmetrical lamp, but the coarse 3D Gaussian representation might be a slightly curved, asymmetrical lamp. Moreover, the initial color generated in Sec. 3.4 may not be consistent with the description of the input text. Faced with these problems, we introduce IP-Adapter to generate multi-view style-consistent images as guidance. First, we propose a transfer mechanism in the structural optimization process, which can effectively guide the structure of the 3D Gaussian representation to align with the input sketch outline. Second, we utilize a straightforward MSE loss to improve the color quality, which can effectively align the 3D Gaussian representation with the input text description. Third, we implement a CLIP-based geometric similarity loss as a constraint to guide the shape towards the input sketch.Multi-view Style-Consistent Images Generation. Due to the rapid and real-time capabilities of Gaussian splatting, acquiring multi-view renderings becomes straightforward. If we can obtain guidance images from these renderings, corresponding to the current viewing angles, they would serve as effective guides for optimization. To achieve this, we introduce the IP-Adapter [51] , which incorporates an additional cross-attention layer for each cross-attention layer in the original U-Net model to insert image features. Given the image features 𝑐 𝑖 , the output of additional crossattention Z is computed as follows:EQUATIONwhere Q = ZW q , K 𝑖 = 𝑐 𝑖 W ′ 𝑘 and V 𝑖 = 𝑐 𝑖 W ′ 𝑣 are the query, key, and values matrices from the image features. Z is the query features, and W ′ 𝑘 and W ′ 𝑣 are the corresponding trainable weight matrices. This enhancement enables us to generate multi-view style-consistent images based on the two image conditions of the reference image and the content images, as shown in Figure 3 . Specifically, herein we use Stable-Diffusion-v1-5 [38] as our diffusion model basis. Given the reference image 𝐼 ref and multi-view splatting images as the content images 𝐼 c , the guidance images 𝐼 g are estimated as follows:EQUATIONwhere 𝑀 is the generator of IP-Adapter, 𝑡 is the sampling time step of inference, and 𝜆 ∈ [0, 1] is a hyper-parameter that determines the control strength of the conditioned content image 𝐼 c . Camera Projection Strategy. As shown in Figure 2 , during the process of Gaussian splatting, our camera projection strategy involves encircling horizontally and vertically. To ensure the stylistic consistency of the generated guidance images, we perform a rotation every 30 degrees for each circle, thereby calculating the guidance images under a progressively changing viewpoint.Structural Optimization. For image-to-3D generation, when selecting the diffusion prior for SDS loss, existing approaches usually use a diffusion model with image as a condition (e.g., Zero-123 [23] ). Differently, we use a diffusion model with text as the condition (e.g., Stable Diffusion [38] ). The reason is that the former does not perform well in generating 3D aspects of the invisible parts of the input image while the latter demonstrates better optimization effects in terms of details and the invisible sections. However, we have to ensure that the reference image plays an important role in the optimization process, so we propose a mechanism of distribution transfer and then use it in subsequent SDS loss calculations. Given guidance images 𝐼 g and splatting images 𝐼 c , the transferred images 𝐼 t are estimated as follows:EQUATIONwhere 𝜇 (•) is the mean operation, 𝜎 (•) is the variance operation. The distribution transformation brought about by the transfer mechanism can bring the distribution of splatting images closer to the distribution of guidance images. In this way, we obtain the transfer image 𝐼 t after distribution migration through guidance image 𝐼 g and splatting image 𝐼 c . To update the 3D Gaussian parameters 𝜃 (𝜇, Σ, 𝑐, 𝛼), we choose to use the publicly available Stable Diffusion [38] as 2D diffusion model prior 𝜙 and compute the gradient of the SDS loss via:EQUATION)where 𝐼 t is the transfer image, y is text prompt, ε𝜙 is similar to Equation 4, 𝑡 is the sampling time step, and 𝐼 s is the input sketch.In conclusion, through the tailored 3D structural guidance, our Sketch3D can mitigate the problem of geometric inconsistencies.Color Optimization. Although through the above structural optimization, we already obtained a 3D Gaussian representation whose geometric structure is highly aligned with the input sketch, some color details still need to be enhanced. To improve the image color quality, we propose to use a simple MSE loss to optimize the 3D Gaussian parameters 𝜃 . We optimize the splatting image 𝐼 c to align with the guidance image 𝐼 g .EQUATION)where 𝜆 linear is the linearly increased weight during optimization, calculated by dividing the current step by the total number of iteration steps. 𝐼 g represents the guidance images obtained from controllable IP-Adapter and 𝐼 c represents the splatting images from 3D Gaussian. The MSE loss is fast to compute and deterministic to optimize, resulting in fast refinement. Note that 𝜆 pose is a parameter that changes with viewing angle, as shown in Figure 2 , in the horizontal rotation perspective, the value of 𝜆 pose is cos(𝜃 azimuth ), in the vertical rotation perspective, the value of 𝜆 pose is 0.3 * cos(𝜃 elevation ). Sketch Similarity Optimization. To ensure that the shape of the sketch can directly guide the optimization of 3D Gaussians, we use the image encoder of CLIP to encode both the sketch and the rendered images, and compute the 𝐿2 distance between intermediate level activations of CLIP. CLIP is trained on various image modalities, enabling it to encode information from both images and sketches, without requiring further training. CLIP encodes highlevel semantic attributes in the last layer since it was trained on both images and text. One intuitive approach involves leveraging CLIP's semantic-level cosine similarity loss to use the sketch as a supervisory signal for the shape of rendered images. However, this form of supervision is quite weak. Therefore, to measure an effective geometric similarity loss between the sketch and rendered image, ensuring that the shape of rendered images is more consistent with the input sketch, we compute the 𝐿2 distance between the mid-level activations [47] of CLIP:EQUATIONwhere 𝜆 sketch is a coefficient that controls the weight, 𝐶𝐿𝐼𝑃 4 (•) is the 𝐶𝐿𝐼𝑃 encoder activation at layer 4. Specifically, we use layer 4 of the ResNet101 CLIP model. 
experiments: 
 In this section, we first introduce the experiment setup in Sec. 4.1, then present qualitative visual results compared with five baselines and report quantitative results in Sec. 4.2. Finally, we carry out ablation and analytical studies to further verify the efficacy of our framework in Sec. 4.3.ShapeNet-Sketch3D Dataset. To evaluate the effectiveness of our method and benefit further research, we have collected a comprehensive dataset comprising 3D objects, synthetic sketches, rendered images, and corresponding textual descriptions, which we call ShapeNet-Sketch3D. It contains object renderings of 10 categories from ShapeNet [3] , and there are 1100 objects in each category. Rendered images from 20 different views of each object are rendered in 512 × 512 resolution. We extract the edge map of each rendered image using a canny edge detector. The textual descriptions corresponding to each object were derived by posing questions to GPT-4-vision about their rendered images, leveraging its advanced capabilities in visual analysis. Currently, there are no datasets available for paired sketches, rendered images, textual descriptions, and 3D objects. Our dataset serves as a valuable resource for research and experimental validation in sketch-to-3D tasks. Implementation Details. In the shape-preserving reference image generation process, we use control-v11p-sd15-canny [57] as our diffusion model 𝐺 2𝐷 . In the Gaussian initialization process, we initialize our Gaussian representation with the 3D diffusion model and utilize Shap-E [12] as our 3D diffusion model 𝐺 3𝐷 . In the multiview style-consistent image generation process, we use the stable diffusion image-to-image pipeline [51] , with a control strength of 0.5. Moreover, we generate two sets of guidance images in two surround modes every 30 steps. In structural optimization, we use stablediffusion-2-1-base [38] . The total training steps are 500. For the 3D Gaussians, the learning rates of position 𝜇 and opacity 𝛼 are 10 -4 and 5 × 10 -2 . The color 𝑐 of the 3D Gaussians is represented by the spherical harmonics (SH) coefficient, with a learning rate of 1.5 × 10 -2 . The covariance of the 3D Gaussians is converted into scaling and rotation for optimization, with learning rates of 5×10 -3 and 10 -3 . We select a fixed camera radius of 3.0, y-axis FOV of 50 degree, with the azimuth in [0, 360] degrees and elevation in [0, 360] degrees. The rendering resolution is 512 × 512 for Gaussian splatting. All our experiments can be completed within 3 minutes on a single NVIDIA RTX 4090 GPU with a batch size of 4.Baselines. We extensively compare our method Sketch3D against five baselines: Sketch2Model [58] , LAS-Diffusion [61] , Shap-E [12] , One-2-3-45 [22] , and DreamGaussian [44] . We do not compare with NeRF-based methods, as they typically require a longer time to generate. Sketch2Model is the pioneering method that explores the generation of 3D meshes from sketches and introduces viewpoint Qualitative Comparisons. Figure 4 displays the qualitative comparison results between our method and the five baselines, while Figure 1 shows novel-view images generated by our method. Sketch3D achieves the best visual results in terms of shape consistency and color generation quality. As illustrated in Figure 4 , the sketch image and the reference image generated in Section 3.3 are chosen as inputs for the latter three baselines. For the same object, the reference image used by the latter three baselines and our method is identical. First, Sketch2Model and LAS-Diffusion only generate shapes and lack color information. Second, Shap-E can generate a rough shape and simple color, but the color details are blurry. Third, One-2-3-45 and DreamGaussian often produce inconsistent shapes and lack color details. All of these results demonstrate the superiority of our method. Additionally, Sketch3D is capable of generating realistic 3D objects in about 3 minutes. Quantitative Comparisons. In Table 1 , we use CLIP similarity [35] and structural similarity index measure (SSIM) to quantitatively evaluate our method. We randomly select 5 objects from each category in our ShapeNet-Sketch3D dataset, choose a random viewpoint for each object, and then average the results across all objects. We calculate the CLIP similarity between the final rendered images and the reference image, as well as between the final rendered images and the text prompt. Moreover, we also calculate the SSIM similarity between the final rendered images and the reference image. The results show that our method can better align with the input sketch shape and correspond to the input textual description.Distribution transfer mechanism in structural optimization. As shown in Figure 5 , the distribution transfer mechanism aligns Figure 4 : Qualitative comparisons between our method and Sketch2Model [58] , LAS-Diffusion [61] , Shap-E [12] , One-2-3-45 [22] and DreamGaussian [44] . The input sketches include sketch images, exterior contour sketches, and hand-drawn sketches. Our method achieves the best visual results regarding shape consistency and color generation quality compared to other methods.the shape more closely with the input sketch, leading to a coherent structure and color. It demonstrates the mechanism's effectiveness in steering the generated shape towards the input sketch.MSE loss in color optimization. As illustrated in Figure 5 , it is evident that the MSE loss contributes to reducing color noise, leading to a smoother overall color appearance. It proves that MSE loss is capable of enhancing the quality of the generated color.CLIP geometric similarity loss in sketch similarity optimization. As shown in Figure 5 , the CLIP geometric similarity loss enables the overall shape to more closely align with the shape of the input sketch. This illustrates that the 𝐿2 loss in the intermediate layers of CLIP can act as a shape constraint.Gaussian initialization through SDS loss. As shown in Figure 6 , we conducted analytical experiments on the Gaussian initialization method to explore which initialization method is better. It can be seen that Gaussian initialization through SDS loss shows good 3D effects only in the visible parts of the input reference image, while problems of blurriness and color saturation exist in the invisible parts. However, the approach of Gaussian initialization with a 3D diffusion model exhibits better realism from all viewing angles. 7 , to explore the fidelity of outcomes generated from the user's freehand sketches, we visualize some of the generated results from hand-drawn sketches. We randomly select three non-artist users to draw three sketches and provide corresponding text prompts. The results show that our method can also achieve good generation quality and consistency for hand-drawn sketches.User Study. We additionally conduct a user study to quantitatively evaluate Sketch3D against four baseline methods (LAS-Diffusion, Shap-E, One-2-3-45, and DreamGaussian). We invite 9 participants and present them with each input and the corresponding 5 generated video results, comprising a total of 10 inputs and the corresponding 50 videos. We ask each participant to rate each video on a scale from 1-5 based on fidelity and consistency criteria. Table 2 shows the results of the user study. Overall, our Sketch3D demonstrates greater fidelity and consistency than the other four baselines.Input "A carrot" "A wooden desk"  
conclusion: 
 In this paper, we propose Sketch3D, a new framework to generate realistic 3D assets with shape aligned to the input sketch and color matching the text prompt. Specifically, we first instantiate the given sketch to the reference image through the shape-preserving generation process. Second, a coarse 3D Gaussian prior is sculpted based on the reference image, and multi-view style-consistent guidance images could be generated using IP-Adapter. Third, we propose three optimization strategies: a structural optimization using a distribution transfer mechanism, a color optimization using a straightforward MSE loss, and a sketch similarity optimization using CLIP geometric similarity loss. Extensive experiments demonstrate that Sketch3D not only has realistic appearances and shapes but also accurately conforms to the given sketch and text prompt. Our Sketch3D is the first attempt to steer the process of sketch-to-3D generation with 3D Gaussian splatting, providing a valuable foundation for future research on sketch-to-3D generation. However, our method also has several limitations. The quality of the reference image depends on the performance of ControlNet, so when the image quality generated by the ControlNet is poor, it will affect our method and impact the overall generation quality. Additionally, for particularly complex or richly detailed sketches, it is difficult to achieve control over the details in the output results.  
sketch3d: style-consistent guidance for sketch-to-3d generation: 
 Anonymous Authors(b) Given a sketch of a lamp and text prompt "A textural wooden lamp", the 3D Gaussians progressively changes throughout the generation process. Our method can complete this generation process in about 3 minutes.prior, and multi-view style-consistent guidance images are generated based on the renderings of the 3D Gaussians. Finally, three strategies are designed to optimize 3D Gaussians, i.e., structural optimization via a distribution transfer mechanism, color optimization with a straightforward MSE loss and sketch similarity optimization with a CLIP-based geometric similarity loss. Extensive visual comparisons and quantitative analysis illustrate the advantage of our Sketch3D in generating realistic 3D assets while preserving consistency with the input.content based on color images. However, several practical scenarios provide only sketches as input due to the unavailability of colorful images. This is particularly true during the preliminary stages of 3D product design, where designers rely heavily on sketches. Despite their simplicity, these sketches are fundamental in capturing the core of the design. Therefore, it is crucial to generate realistic 3D assets according to the sketches.Inspired by this practical demand, studies [14, 25, 40, 58] have endeavored to employ deep learning techniques in generating 3D shapes from sketches. Sketch2model [58] employs a view-aware generation architecture, enabling explicit conditioning of the generation process based on viewpoints. SketchSampler [6] proposed a sketch translator module to exploit the spatial information in a sketch and generate a 3D point cloud conforming to the shape of the sketch. Furthermore, recent works have explored the generation or editing of 3D assets containing color through sketches. Several [33, 50] proposed a sketch-guided method for colored point cloud generation, while others [19, 26] proposed a 3D editing technique to edit a NeRF based on input sketches. Despite these research advancements, there are still limitations hindering their widespread applications. First, generating 3D shapes from sketches typically lacks color information and requires training on extensive datasets. However, the trained models are often limited to generating shapes within a single category. Second, the 3D assets produced through sketch-guided generation or editing techniques often lack realism and the process is time-consuming.These challenges inspire us to consider: Is there a method to generate 3D assets where the shape aligns with the input sketch while the color corresponds to the textual description? To address these shortcomings, we introduce Sketch3D, an innovative framework designed to produce lifelike 3D assets. These assets exhibit shapes that conform to input sketches while accurately matching colors described in the text. Concretely, a reference image is first generated via a shape-preserving image generation process. Then, we initialize a coarse 3D prior using 3D Gaussian Splatting [52] , which comprises a rough geometric shape and a simple color. Subsequently, multiview style-consistent guidance images can be generated using the IP-Adapter [51] . Finally, we propose three strategies to optimize 3D Gaussians: structural optimization with a distribution transfer mechanism, color optimization using a straightforward MSE loss and sketch similarity optimization with a CLIP-based geometric similarity loss. Specifically, the distribution transfer mechanism is employed within the SDS loss of the text-conditioned diffusion model, enabling the optimization process to integrate both the sketch and text information effectively. Furthermore, we formulate a reasonable camera viewpoint strategy to enhance color details via the ℓ 2 -norm loss function. Additionally, we compute the 𝐿2 distance between the mid-level activations of CLIP. As Figure 1 shows, our Sketch3D provides visualization results consistent with the input sketch and the textual description in just 3 minutes. These assets are readily integrable into software such as Unreal Engine and Unity, facilitating rapid application deployment.To assess the performance of our method on sketches and inspire future research, we collect a ShapeNet-Sketch3D dataset based on the ShapeNet dataset [3] . Considerable experiments and analysis validate the effectiveness of our framework in generating 3D assets that maintain geometric consistency with the input sketch, while the color aligns with the textual description. Our contributions can be summarized as follows:• We propose Sketch3D, a novel framework to generate realistic 3D assets with shape aligned to the input sketch and color matching the text prompt. To the best of our knowledge, this is the first attempt to steer the process of sketch-to-3D generation using a text prompt with 3D Gaussian splatting. Additionally, we have developed a dataset, named ShapeNet-Sketch3D, specifically tailored for research on sketch-to-3D tasks.• We leverage IP-Adapter to generate multi-view style-consistent images and three optimization strategies are designed: a structural optimization using a distribution transfer mechanism, a color optimization with ℓ 2 -norm loss function, and a sketch similarity optimization using CLIP geometric similarity loss. • Extensive qualitative and quantitative experiments demonstrate that our Sketch3D not only has convincing appearances and shapes but also accurately conforms to the given sketch image and text prompt. 
 
 Appendix:  


Decision: This paper presents a new model, Sketch3D, designed for the task of sketch-text guided 3D content generation. The proposed model synergizes 3D Gaussian Splatting, ControlNet, and IP-Adapter, addressing the limitations of previous methods in terms of fidelity and consistency. Although one reviewer expressed concerns about its user study and comparison with Nerf-based methods, the authors appear to have addressed these issues in their rebuttal. Besides, the work was recognized as pioneering in incorporating 3D Gaussian Splatting into sketch-guided 3D generation, and the results are impressive. Therefore, I recommend accepting this paper.

Reviews: review: paper_summary: paper_summary: Sketch3D has developed a pioneering framework designed to create realistic 3D assets. This framework aligns the shape of the generated 3D assets with an input sketch, while also ensuring the color matches a provided text prompt. Furthermore, the researchers have compiled a specialized dataset, ShapeNet-Sketch3D, which will facilitate research in the sketch-to-3D domain.
strength_weakness: strength_weakness: - the first attempt to steer the process of sketch-to-3D generation using a text prompt with 3D Gaussian splatting.
- 3D Gaussian initialization approach is resonable.
- reasonable optimization including structural optimization, color optimization and sketch similarity optimization.
- good results.
limitations: limitations: - Table1. How do you get the rendered images for Sketch2Model and LAS-Diffusion which lack textures?
- L630: does 3min include the initialization?
- Figure 6 shows initialization results of different results, could you please show the final results of these objects?
paper_summary: paper_summary: The paper presents a sketch-text guided 3D content generation method to tackle the limitations of existing solutions in terms of fidelity and consistency. The proposed Sketch 3D first generates a reference image guided by a pair of sketch-text prompt for a coarse 3D prior estimation using Gaussian Splatting, and then refines the 3D content with three newly developed optimisation stategies. Both qualitative and quantitative experiments were carried out to demonstrate the effectiveness of the proposed method.
strength_weakness: strength_weakness: Overall, the idea introduced in this work is interesting and is one of the pioneering attempts that leverage multimodal prompts and recent advance in computer graphics to deal with the challenges in 3D AIGC.
limitations: limitations: However, important details about the experiments are missing, and this makes the conclusion less convincing.

In Sec. 4.1, the author(s) claim that they did not compare NeRF-based methods since they are typically time-consuming. It would be interesting to see comparisons  between NeRF-based methods and the proposed one with regard to both quality and efficiency.

In Sec. 4.3, a user study was conducted to evaluate the superiority of the proposed method against four baseline methods. Unfortunately, the background of the 9 participants involved in the study and more details about the evaluation process are missing. Without sufficient details, the results in Table 2 are not convincing.
paper_summary: paper_summary: The paper presents Sketch3D, a novel framework designed to generate realistic 3D assets based on input sketches and textual descriptions. The approach addresses the limitations of existing sketch-to-3D generation methods, which often lack color information and require extensive datasets. Sketch3D leverages 3D Gaussian Splatting, ControlNet for shape-preserving reference image generation, and IP-Adapter for multi-view style-consistent image generation. The framework incorporates three optimization strategies: structural optimization via distribution transfer, color optimization with MSE loss, and sketch similarity optimization using CLIP-based geometric similarity loss. The effectiveness of Sketch3D is demonstrated through extensive visual comparisons and quantitative analysis, showing superior performance in generating lifelike 3D assets consistent with input sketches and textual descriptions.
strength_weakness: strength_weakness: + The combination of 3D Gaussian Splatting, ControlNet, and IP-Adapter introduces a novel way to generate 3D assets from sketches and text, addressing key limitations of previous methods.
+ Dataset Contribution: The introduction of the ShapeNet-Sketch3D dataset provides a valuable resource for future research in sketch-to-3D generation tasks.
Extensive Evaluation: The paper includes comprehensive experiments, ablation studies, and user studies to validate the proposed method, offering a thorough assessment of its strengths and limitations.
limitations: limitations: - Dependency on ControlNet: The quality of the reference images generated by ControlNet significantly impacts the overall performance of Sketch3D. Poor image quality from ControlNet can degrade the final 3D asset generation.
- Complexity in Detailed Sketches: For highly detailed or complex sketches, the method may struggle to maintain control over the finer details in the output, potentially reducing the fidelity of the generated assets.
- Limited Generalization: While the method performs well on the provided dataset, its generalization to more diverse and unstructured sketches or textual descriptions needs further exploration.
- Lack of Complex Text and Object Experiments: The paper lacks experimental results involving complex text and objects, such as "a fox", "a dog reading a book", and "an astronaut riding a horse". Including such examples would better demonstrate the model's capabilities and limitations in handling more intricate scenarios.
paper_summary: paper_summary: This paper proposes a novel generation paradigm Sketch3D to generate realistic 3D assets with shape aligned with the input sketch and color matching the textual description. Their contributions can
be summarized as follows:
• Authors propose Sketch3D, a novel framework to generate realistic 3D assets with shape aligned to the input sketch and color matching the text prompt. To the best of our knowledge, this is the first attempt to steer the process of sketch-to-3D generation using a text prompt with 3D Gaussian splatting. Additionally, we have developed a dataset, named ShapeNet-Sketch3D, specifically tailored for research on sketch-to-3D tasks.
• Authors leverage IP-Adapter to generate multi-view style-consistent images and three optimization strategies are designed: a structural optimization using a distribution transfer mechanism, a color optimization with ℓ2-norm loss function, and a sketch similarity optimization using CLIP geometric similarity loss.
• Extensive qualitative and quantitative experiments demonstrate that their Sketch3D not only has convincing appearances and shapes but also accurately conforms to the given sketch image and text prompt.
strength_weakness: strength_weakness: 1. Gaussian work brings new research potential to 3D generation. The author innovatively combines sketching with 3D generation tasks through Gaussian.
2. Smooth writing and clear logic
3. The illustrations are clear and easy to understand.
limitations: limitations: 1. Although the author's effect looks good, I feel that there are some limitations in the 3D generation work of introducing Gaussian, and it does not realize the research potential of Gaussian. For example, the wings on both sides, the corners of the round table, and the cherry platycodon in Figure 4 are not very effective.
2. MSE should constrain the color so that the color of the generated 3D is similar to the color of the generated image. However, I did not see the outstanding effect of MSE. The color of the lampshade does not seem to be consistent with the color of the image. I also did not see the effect of MSE in the visualization results of ablation loss. If the author could provide me with more results, I might have a new perspective.
3. The author uses the diffusion model to generate images aligned with the sketch as constraints for 3D generation. I understand that the 3D model will definitely undergo some changes during the subsequent optimization process (Figure 6 and Figure 7 above), but I want to know if this change process is uncontrollable? (Points appeared above the house, wine glasses were uneven, table corners were blunted) I think I would be happy to adjust my rating if it was controllable.