Diff3DS: Generating View-Consistent 3D Sketch via Differentiable Curve Rendering
Authors: Yibo Zhang
Lihong Wang
Changqing Zou
Tieru Wu
Rui Ma
Venue: ICLR 2025 Poster (2025)
Keywords: 3D Sketch
Sketch Generation
Diffusion Models
Primary area: generative models
TL;DR: In this paper, we propose Diff3DS, a novel differentiable rendering framework for generating view-consistent 3D sketch from flexible inputs such as a single image or text.

Abstract: 3D sketches are widely used for visually representing the 3D shape and structure of objects or scenes. However, the creation of 3D sketch often requires users to possess professional artistic skills. Existing research efforts primarily focus on enhancing the ability of interactive sketch generation in 3D virtual systems. In this work, we propose Diff3DS, a novel differentiable rendering framework for generating view-consistent 3D sketch by optimizing 3D parametric curves under various supervisions. Specifically, we perform perspective projection to render the 3D rational Bézier curves into 2D curves, which are subsequently converted to a 2D raster image via our customized differentiable rasterizer. Our framework bridges the domains of 3D sketch and raster image, achieving end-to-end optimization of 3D sketch through gradients computed in the 2D image domain. Our Diff3DS can enable a series of novel 3D sketch generation tasks, including text-to-3D sketch and image-to-3D sketch, supported by the popular distillation-based supervision, such as Score Distillation Sampling (SDS). Extensive experiments have yielded promising results and demonstrated the potential of our framework. Project: https://yiboz2001.github.io/Diff3DS/

Published as a conference paper at ICLR 2025
DIFF3DS: GENERATING VIEW-CONSISTENT 3D SKETCH VIA DIFFERENTIABLE CURVE RENDERING
Yibo Zhang' Lihong Wang' Changqing Zou Tieru Wu
Rui Ma Ma 1.4+
Jilin University State Key Lab CAD&CG, Zhejiang University Zhejiang Lab Engineering Research Center of Knowledge-Drive Driven (ybzhang23, w1h23/ @ma ils. jlu. aat.edd.com en (wutr, ruim @ jlu. edu. cn aaronzoul 25 gma i 1. com
Intelligence, MOE, China
Figure 1: In this paper, we propose Dift3DS, a novel differentiable rendering framework for generating view-consistent 3D sketch from flexible inputs such as a single image or text.
ABSTRACT
3D sketches widely used for visually representing the 3D shape and structure of objects or scenes. However, the creation of 3D sketch often requires users possess professional artistic skills. Existing research efforts primarily focus enhancing the ability of interactive sketch generation in 3D virtual systems. this work, we propose Diff3DS, novel differentiable rendering framework for generating view-consistent 3D sketch by optimizing 3D parametric curves under various supervisions. Specifically, we perform perspective projection to render the 3D rational Bézier curves into 2D curves, which subsequently converted to a 2D) raster image via our customized differentiable rasterizer. Our framework bridges the domains of 3D sketch and raster image, achieving end- to-end optimization of 3D sketch through gradients computed in the 2D image domain. Diff3DS can enable series of novel 3D sketch generation tasks, including text-to-3D sketch and image-to-33 3D sketch, supported by the popular distillation- based supervision, such as Score Distillation Sampling. Extensive experiments have yielded promising results and demonstrated the potential of our framework. . github. co/Diff3DS
Project page is at penttps://yiboaee
INTRODUCTION
3D sketches, which utilize strokes to emphasize abstraction and visually encapsulate the 3D shape and structure objects or scenes, serve as indispensable tools for visualizing concepts and ideas. Existing research efforts have primarily focused on exploring how enable interactive creation of 3D sketch
"Corresponding author.

---

Published as a conference paper at ICLR 2025
in 3D virtual systems (Wesche & Seidel 2001 Grimm & Joshi 2012: Bae al., 2008 and immersive
environments ( Yu et al. |02 Xii et al. 2008; Kim et al. 2023 Jiang et al. [022]] Kim & Bae
2016 Yu et al. 202 lb Arora & Singh 2021 Yue al. (017)) Such interactive 3D sketch creation
tools provide artists with unprecedented freedom, allowing them to draw their inspirations directly in the immersive 3D environment. However, it usually requires users to possess professional artistic skills and experience for 3D operation, making the 3D sketch creation less friendly for ordinary users. There is a notable absence of a user-friendly 3D sketch generation method in the community. Moreover, in the area of wire art generation, 3D sketch or 3D curves are also widely studied to abstract the desired visual concepts from diverse inputs, e.g.. 3D surfaces (Yang et al. 2021), multi-view images (Liu et al., 2017) and text (Tojo et al. 2024 Qu et al. 2024). These representations provide essential shape and structure of the artwork as well as hold significant potential as intermediate formats for tasks such as 3D reconstruction. However, generating view-consistent 3D sketches from flexible inputs, such as text or a single image, remains largely unexplored.
In recent years, content reconstruction or generation via differentiable rendering has gained great attention in the fields of computer graphics and vision research. The goal is to convert the traditional rendering pipeline into differentiable image synthesis process and employ the 2D image supervision to optimize the object or scene representation. For example, NeRF (Mildenhall et al. 2021) bridges the 3D neural representation and raster image domain with a differentiable volume rendering pipeline, and has contributed to the further development of the 3D generation (Mohammad Khalid et al. [022]
Tsalicoglou et al.] 2024 Jain et al. 2022 Poole et al. [023] Lin et al., (002) by leveraging the
continuous advancements in multimodal supervision (Radford et al. 202 Poole et al. (023)). For
the generation of vectorized or parametric content, differentiable rendering has also been investigated
(Li et al. 2020 [Schaldenbrand et al. 2022 Frans et al. 2022 Jain et al. 2023
Xing et al. 2023 Qu
ct al. [2023] [ing et 20224] Banerjee et al. 20044] Vinker et al. 2022 2023
Thamizharasan et al.
2024b a). One representative work is DifVVG (Li et al. 2022)), which proposes differentiable 2D
vector graphics rasterizer to compute gradients from raster images and optimize the vectorized image (e.g.. 2D Bézier curves and other parametric shapes) via puck-propagation. For generation of sketch, CLIPasso generates 2D sketches using DiffVG to optimize the parameters of the 2D Bézier curves directly with respect to CLIP-based perceptual loss. Recently, 3Doodle (Choi et al. 2024) generates view-consistent 3D sketches of the target object by directly optimizing the parameters of 3D strokes to minimize the perceptual losses for given multi-view images. It also proposes differentiable 3D curve rendering pipeline which integrates DifTVG as key component. Although 3Doodle achieves promising results for generating 3D sketches, its optimization requires supervision from multiview images, which makes it more like a reconstruction pipeline instead of more flexible generation pipeline. Meanwhile, its performance is constrained by the inherent limitations of DifTVG's original design. It relies on an approximate perspective projection to obtain the 2D sketch rendering and ignores the depth order of the curves, which may lead to color conflicts in the 3D sketch and limit its effectiveness more complex tasks like colored sketch generation.
In this paper, we present Diff3DS, novel differentiable rendering framework for generating view- consistent 3D sketch from flexible inputs such as a single image or text. Specifically, we represent the 3D sketch as a set of 3D rational Bézier curves and perform the perspective projection to obtain the 2D rational Bézier curves. Then, we propose new differentiable rasterizer based on DiffVG to accurately render the projected 2D curves based on the depth order, so that more accurate occlusion relationships between curves can be modeled, especially for colored curves. Our framework supports end-to-end optimization of 3D curve primitives under flexible supervisions such as the distillation- based loss. By employing the recent Score Distillation Sampling algorithm distill prior knowledge from pre-trained 2D image generation model, we achieve 3D sketch generation from the text or single image input. We conduct quantitative and qualitative comparisons between Diff3DS and related methods. The results demonstrate the superiority of our approach in generating view-conse niw-coosiistent 3D sketches. Ablation studies further validate the effectiveness of the key components of our method. Additionally, analysis of the rasterizer shows its potential for generating colored sketches.
We summarize our contributions as follows: (1) We propose DiffDDS, novel differentiable rendering framework for generating view-consistent 3D sketch from flexible inputs such as a single image or text. (2) We are the first to represent the 3D curve as 3D rational Bézier curve and design a depth-aware rasterizer that can enable precise and differentiable rendering of both black and colored 3D curves. (3) We conduct comprehensive experiments on novel text-to-33 to-3 and image-to-3D sketch generation tasks and the results demonstrate the superiority of our method.
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORKS
3D Sketch Generation
Existing research on 3D sketch generation mainly focus on interactive
artistic sketch creation within 3D virtual systems |Wesche & Seidel 2001: Grimm & Joshi 2012
Bac et al., 2008), and immersive environments YYu et al. [021a Xin et al., 2008 Jiang et al. 2021
Kim & Bae 2016; Yu et al. 202Tb Arora & Singh 2021). However, these works usually require users to possess professional artistic skills and 3DD operation experience, making it less user-friendly for ordinary users. The recent 3Doodle (Choi et al. 2024) is able to generate expressive 3D sketches from multiview image observations, but it is more like multíview reconstruction instead of generation and the quality of the generated results highly depends on the number of observation viewpoints. By distilling prior knowledge from pre- pre-trained 2D image generation models using methods like Score Distillation Sampling (Poole al., 2023 Liu al., 2023). we achieve user-friendly and view-consistent 3D sketch generation from flexible text or single image input.
Differentiable Rendering of Curves There have been several works for differentiable rendering of curves. DifFVG proposes a differentiable rasterizer for the creation of vector graphics represented by 2D curves and shapes. DRPG (Worchel & Alexa 2023) investigates differentiable rendering of 3D parametric curves and surfaces, by piecewisely approximating the continuous parametric representations with a triangle mesh. 3Doodle (Choi et al. 2024) shares a similar differentiable rendering pipeline with us, including projecting the 3D Bézier curves to 2D Bézier curves, which are then rendered by the differentiable rasterizer from DiffVG. However, 3Doodle approximates the perspective projection an orthographic projection obtain 2D projected curves and ignores the depth ordering between the curves. Instead, we perform perspective projection to the 3D rational Bézier curves and designs new DiffVG-based ratersizer which can maintain the depth ordering between the projected 2D rational Bézier curves.
Multimodal-Drivee 3D Content Generation Motivated by the success of the text-to-image research, pioneering works (Poole et al. 2023 Wang et al., 2023a) introduce the Score Distillation Sampling (SDS) algorithm and leverage the pre-trained text-to-image models as prior knowledge to optimize 3D representations. Subsequent works combine SDS with various differentiable 3D representations to explore the capabilities of text-to-3D object generation, such as DMTet (Lin)
et al. 2023 Chen et al. (202) and 3D Gaussians (Tng et al. 2024: Yi et al. (024).. Furthermore, Zero- 1to-33 Liu et al. (002) proposes an saange-giieed SDS algorithm that distills the 3D-consistent priors to optimize 31D representations given the input image, and its pipeline inspires the development of subsequent image-to-33 object methods (Liu et al.) 2023 Qian et al., 2024 Wang & Shi] 2023)
Wu et al. 2023 Sta 2023). Distinguishing from prior endeavors, our work stands out by focusing on
generating view-consistent 3D sketch by optimizing 3D parametric curve primitives.
3 PRELIMINARY
DiffVG (Li al. 2020) proposes differentiable rasterizer for vector graphics that supports converting 2D Bézier curves to the raster image domain and proppparpggg the gradient for optimization. Given the 2D curves parameters O, the vector graphics scene is defined as f(z, y, θ), and the raster image is defined as the 2D grid sampling over the space of f (r. y. θ). To compute the color of a 2D location (a,, y) E RR, DiffVG utilizes the inside-outside test (Neh) find the curves overlapped with the location initially. Subsequently, it sorts them according to a user-specified order and calculates the color using alpha blending (Porter& & Duff 1984). Due to the inside-outside test, the scene functioon f is not differentiable with respect to curve parameters. Thus, it uses the anti-aliasing technique to make the pixel color differentiable. By prefiltering f over convolution kernel with support A, sampling I [xx y) at pixels can yield an alias-free image:
I(x v) = Js. k(u, f( - u; O)
Due to the integration over the filter support region, the focus shifts from the color value at the center point to the average pixel color. The continuous change in the average color induced by the curve movements makes the function T(x, v) differentiable.
3

---

Published as a conference paper at ICLR 2025
The goal is to compute the gradients of with respect to O:
@I y) Ô
de //, k(u, fff - - d.d
(2)
ae
DiffVG proposes two approaches Monte Carlo Sampling and Analytical Prefiltering to evaluate the pixel integral, which does not have closed-fore solution in general. We focus the first approach which performs better. The pixel integral can be discretized with Monte Carlo sampling (Ganacim
al. 2014)
u,y u,y T(, v) = //. -
1
du de
E E(uu, vz) U1 y- 04; +) N
but geometric discontinuities hinder the interchangeability of the integral and differential operators, which consequently obstructs the direct representation of 01 86 as a discretizable integral. Thus,
rewrites the pixel color the sum of integrals over multiple sub-regions A,, and applies the Reynolds transport theorem to handle all the discontinuous changes on the boundary:
(I v) 00
a -/ s(u, (u, v) du du + L A.(0)
(Do(t)
nn(t) g(p(t)) dp(t),
=
(4)
where g(u, v) is the multiplication of the scene function f and kernel for brevity, 0A, is the boundary of area A,, n(t) the outward normal of AA and p(t) denotes the 2D points the A The first integral is responsible for the differentiation of color and the transparency, while the second term is responsible for the change of the boundaries, and the boundary integral can be estimated by the Monte Carlo estimator as:
1 i E/ (Vep (e)p((()((())))))))))))))))))))) dp(()
(Vep; n,) (g(p, + enj) - p(,, - eny))
N
P(p,
(5)
where is a small number and P(p) denotes the probability density of sampling the curve c and point Pj the boundaries.
4
DIFF3DS
4.1
OVERVIEW
We propose Diff3DS, a novel differentiable rendering framework for generating view-consistent 3D sketch by optimizing 3D parametric curves. A 3D sketch is defined as collection of 3D parametric curve strokes that reside in 3D space. We particularly focus the 3D rational Bézier curve and our derivation supports the 3D linear, quadratic, and cubic rational Bézier curves. The theoretical trainable parameters include the control point position, curve width, and rgb-alpha color. Follow the previous methods (Vinker et al.) 2022 2023 Choi et al.) 2024), we represent each stroke by 3D cubic Bézier curve and only optimize the control point position in our current implementation. Moreover, weights for rational Bézier curves are disregarded to simplify the computation. Fig. illustrates the pipeline.
DiffDD designed with three stages. First, to render the 3D rational Bézier curve strokes θ at given viewpoint, we perspectively project these curves onto the camera plane (Sec. 4.2p. p. Then, we render the projected 2D rational Bézier curves to the raster image using a customized differentiable rasterizer. (Sec. [.3). Finally, we will perform mack-propagation with the gradient computed from the rendered image to optimize the 3D sketch parameters (Sec. (.4).
4.2 3D RATIONAL BÉZIER CURVE PROJECTION
In this section, we primarily explain how to project the 3D rational Bézier curves in camera coordinates onto the 2D camera focal plane, and detailed proof is provided in Appendix Given the control points P. (P. Py P.)) E R the associated non-negative weights w; and the Bernstein polynomials B (t), for 0 < < 1, define the 3D rational Bézier curve p(t) of degree by:
p(t) Σ B (t)w;
P = (t)))
(6)
1=0 Eド= Ban
4

---

Published as a conference paper at ICLR 2025
Assume the focal plane is f, where is the focal length, the corresponding projected 2D curve p(t) under the pinhole camera perspective projection can be defined as:
(()) - ((((),(0)) = m(t) J,
7(t) 2(1) Π)
Due to rational Bézier curves are projective invariant, the curve p(t) still is identical to the 2D rational Bézier curve defined by projected control points. Given projected 2D control points P, =
(P, P!
p
f:
p f) R² and adjusted weights wA P, p(t) can be written as following:
p.
n1
Bin (t)ws P. - (x()).
p(t)
(8)
im Z==z Ban(t)w,
4.3 2D RATIONAL BÉZIER CURVES DIFFERENTIABLE RASTERIZATION
This section primarily focuses on the differentiable rasterization of 2D curves θ obtained through projection. To render the projected rational Bézier curves and maintain their occlusion relationship in 3D space, especially for colored curves, we customize new differentiable rasterizer based on DiffVG (Li et al. 2020). Given a 2D pixel location (z,, y) and projected curves O, we first employ a inside-outside test to identify all the overlapping curves with this pixel location. Then, we sort the overlapping curve points and calculate the color of this pixel location using alpha blending algorithm.
The inside-outside test identifies curve as overlapping with a location if the closest distance between them is less than half of the curve width, and the closest point to the location are considered as the overlapping point. However, the identification is challenging. For the reth-degree rational curve p(t), computing the f that minimizes the distance (p(t) - q) where q = (z, y) equals to solve the roots of polynomial with degree 3n 2. For the cubic rational curve, 7th order polynomial needs to be solved, which does not have a flosed-form solution. Inspired by DifFVG, we solve the polynomial using bisection and the Newton-Raphson method (William H. Press & Flannery,
2007). The iterative
solver obtains its initial guess from isolator polynomials (Sederberg & Chang. 1994) the real roots
of 7th order polynomial can be isolated by the roots of two cubic auxiliary pol polynomials. Please refer to Appendix C for more details.
The rasterized result should faithfully maintains the occlusion in 3D space. our task, different points along the same projected 2D curve may have different depths. Therefore, each point should be assigned with its own specific order for the color blending process. the original DifFVG, each curve is initialized with user-specific order, meaning all the points along the same curve share this uniform order. In contrast, our sort order is determined by the depth order among curve primitives, For each overlapping point, we compute its unprojected z-depth in the 3D space using the corresponding 2D projected curve control points, and employ it as the primary sort order. To mitigate the z-fighting issue caused by floating footing-point errors, also utilize user-specified order the secondary order. After obtaining the 2D curve scene function f(x y, +). the rendered raster image I will be calculated using the Monte Carlo Sampling strategy (Sec. 3 with Eq. (3). Fig. 2 shows an example. Our rasterizer faithfully renders the colored curves and maintains occlusions according to the depth order.
A rasterized result contains quadratic rational Bézier curves and line. All curves share the same control point posi- tions in pixel space and the depths but with different weights. Our rasterizer faithfully renders the curves and maintains occlusions according to the depth order. (c.g., The upper half of each curve has a greater depth than the line, while the lower half has a lesser depth. This difference results in varying color blending outcomes at the overlapping regions).
Figure 2:
[1805, 513) (1023, 180%))
Point Weeghte: 1:3)
Point Depthe [1A 1.0
Curse
Pount [oses,
Point Wegass:
Point Deptis
Curve C
Control Points [10055 512) (1023, J0EN)
Prine Weigite [10,41 Point Deptic (50,25, 25,11) 1.0]
Line D
Conare Poiats [[511,.00 01, (511A, 1023g]
Print Weigans 110
(600 THO)
Paint Nepthe (2..5
5

---

Published as a conference paper at ICLR 2025
SD Skerch
Busiet Image /
Naiay Image &
Pacado GT 4
Diff.
Prestrineed Daffacom Morkel
Rendener
[Sec
Gradiant
Figure 3: We generate the 3D sketch Ô, represented as a set of 3D strokes, from the text or image input. We render the raster image 7 from the random camera via differentiable renderer (Sec. (4). Then, the pre-trained diffusion model, conditioned on the input, diffuses the rendering / and predicts the pseudo ground truth To. The discrepancies between Io and I are used to update the 3D sketch.
4.4 GRADIENTS BACK-PROPAGATION
The rendering framework is fully differentiable since the projection and rasterizer are differentiable. Thus, the gradients of I with respect to 3D curves parameter θ can be calculated with the chain rule: ØT (a. v) (r [rr, y) 00 where denotes the gradients of 7 with respect to 2D projected curves a
ar
ae
0
ae
ie
+ that be computed using Eq. (4) and Eq. (),
and
denotes the gradients of projected 2D ae
curves θ with respect to 3D curves Ô. 0I(x, y) calculates the gradients of control point positions,
00
color, and width for projected 2D eurves. The position gradients 3D curve control points are calculated with 0e
which is based on the gradients of the projected 2D control points. The color a
and width gradients of 3D curves directly inherit the corresponding gradients of the projected 2D curves.
MULTIMODAL-DRIVEN 3D SKETCH GENERATION
Diff3DS supports end-to-end optimization of 3D sketch under flexible supervisions. By employing the Score Distillation Sampling algorithm, we propose the promising tasks text-to-3D sketch and image-to-3D sketch, with the goal of generating 3D sketch from flexible text or single image input.
5.1 TEXT-TO-3D SKETCH
Pioneering works (Poole et al. 2023 Wang al., 2023a propose the Score Distillation Sampling (SDS) algorithm, utilizing the prior of pre-trained 2D text-to-image model to optimize 3D represen- tations. By integrating SDS with Dift3DS, we distill the diffusson prior to generate the 3D sketch Ô from the input text. Given a pre-trained diffusion model tg. text embedding y, rendered image with our rasterizer R. and the noise timestep I, image / will be added with noise to obtain noisy image I; 4 = I + vI- āye, where e ~ NO,0, I) and ā is the cumulative product scaling at timestep t. Then the gradient of SDS loss on θ is given by:
0T
VoLsDs R(ô)) = E,,e ((t)(e((T a
(9)
where (t) is a weight based on the timestep t, and the predicted noise sampled by classifier-fre guidance (CFG) (Ho & Salimans 2021) with weight À denotes as:
é 1,1,,,440 () e (Ir; 0, + X(Ea v, e (L;
(10)
By deriving the pseudo ground truth image i with one-step denoising:
i0
Vo (1. vT aré (Tsss
(11)
we can transform Eq. into an equivalent form as follows:
Vou
10) 01
V VoLeDS( - = F wwt)
(12)
Vor
0

---

Published as a conference paper at ICLR 2025
which can be regarded matching the input view I with predicted lo-
Similar with (Poole et al. 2023), due to the bias of 2D text-to-image model, encounter seri- ous view inconsistency problem known as the Janus problem. This problem persists when directional prompts (e.g.. "the front view of...") are used to distinguish between different views. To mitigate this issue, we integrate MVDream (Shi et al. 2024) with our framework, which is able to generate 3D-aware four view images using input text and camera poses.
A-1
A=
4-7.5
-030
50
-100
-500
Figure 4: Pseudo ground truth lo visualization. As the CFG weight decreases, the effective supervi- sion region of To also decreases.
The impact of the CFG weight À on the optimiza- tion process has received attention existing text-to-3D methods. By visualizing the pseudo ground truth in after denoising with different levels of weight À at large noise timesteps in Fig. we find that as À decreases, the effective supervision region of lo decreases, hereby impacting the final shape of results.
5.2 IMAGE-TO-31 SKETCH
Zero-I-to-3 (Liu ct al. 2023) proposes image-guided SDS method that conditions the input view image 7 and relative camera extrinsic (R, T), based on its view-conditioned model. Our framework is able to generate 3D sketch è from the input reference image by distilling the 3D consistent prior:
01 VoLaDS(é 7= R(Ō)) Er,, (T)) R,1, -)) 0
(13)
5.3 DYNAMIC NOISE DELETION
We find that our framework is affected by the issue of gradient sparsity the Monte Carlo Sampling strategy. In Reynolds' formula, the gradient from pixel space is determined by the curve boundaries. This results in the loss of information from pixels that are not incident to those boundaries. where significant displacements of curves required, the optimization tends to get trapped in local optimum state: curves contract the boundaries to minimize the loss function. During our training process, some curves tend contract extremely small lengths noises. To obtain clean results, we propose the Dynamic Noise Deletion strategy. We discretize each curve into n line segments (n 20) and accumulate their lengths the total length of the curve. The curve with lengths below a specified threshold will be considered as noise and removed dynamically.
cases
5.4 TIME ANNEALING SCHEDULE
To further enhance the performance of the results, we employ a time annealing schedule similar with (Wang et al. 20236 Zhu al. 2024 Shi et al. 2024 HHang et al. 2024). During the optimization process, we gradually adjust the maximum and minimum time step for SDS in linear manner. The large noise timestep focuses on aligning the curves with the semantic content of the input text or image during the initial training phase. Conversely, the small noise timestep employed during the later training phase focuses on further enhancing the details.
EXPERIMENTS
6.1 IMPLEMENTATION DETAILS
We implement rendering framework in C++/UUDA with a PyTorch interface (Paszke et al. 2019). In the experiment, a user-specified number of curves will be randomly initialized within a sphere of radius 1.5. We randomly sample the camera position using the radius from 1.8 to 2.0, with the azimuth in the range of 180 to 180 degrees, the elevation in the range to 30 degrees and the field of view (fov) of 60 degrees. For the pre-trained model, we apply Stable Diffusion 2.1 (sta) and
7

---

Published as a conference paper at ICLR 2025
"A DSLR photo realistic lighthouse, [suffix]]
"A fiee phoenix, mythical bird, engulfed in flames, [suffix]]
"Higly detailed, majestic royal ship, realistic puinting.... [sulfix]]" DiffDD (Ours)
DifSkktthhr
DreamGaussian
MVDream
Figure Qualitative results of the text-to-3D sketch task. Existing text-to-3D 3D methods fail to generate sketch-style results, even with the addition of the "sketch in black and white, line drawing" suffix.
Table 1: Evaluation of text-to-3D sketch task.
Table 2: Evaluation of image-to-3D sketch task.
CL.I--ccore"
Novel Views
Reference View
Method
BI.IP-Score
Method
CLIP--Soore
VIT B/16
VIT B/32
LPIPS
ViT B/16
VIT B/32
DreamGaussian
0.2561
0.2453
0.4990
NEF
0.6612
0.6605
0.3997
MVDream
0.2763
0,2653
0.4949
3Doodle
0.6746
0.2567
Diff3DS (Ours)
0.3046
0.3034
0.5038
Dif33D (Ours)
0.6768
0.6846
0,2647
MVDream (Shi et al. 2024) for the text-to-3D sketch task. And apply Zero-1-to-3 (Liu et al.
2023) and Stable-Zero 23 (Sta 2023) for the image-to-3 sketch task. The training process requires 1 hour for the text-to-3I sketch task and 2 hours for the image-to-3D sketch task single NVIDIA
A10 GPU. More details can be found in the Appendix A
6.2 TEXT-TO-3D SKETCH
To the best of our knowledge, we are the first text-to-3D t0D sketch method. We select the existing text-to-3D object methods DreamGaussian (Tang et al., 2024) and MVDream (Shi et al. 2024) as compared baselines, and further choose the text-to-2D sketch method DiffSketcher (Xing as the additional perceptual reference.
Baselines
et al. 2023
Qualitative Comparisons Fig. 5 shows the qualitative results of the text-to-3D sketch task. We compared the 3D results of the text-to-3D -t baseline method with method, and further provide the 2D results of DiffSketcher. To encourage baseline methods to generate sketch-style results, we append the suffix "sketch in black and white, line drawing" " to each prompt inputted the baselines. From the results, it can be seen, even with the additional suffix, the general text-to-3D baselines DreamGaussian and MVDream cannot generate the sketch-style content. For DiffSketcher, the results are more like sketch draws and only in 2D. In contrast, our method can generate 3D view-consistent sketches with clearly defined curves.
Quantitative Comparisons We collect text prompts from previous works (Poole al. 2023 Shi et al. 2024) and websites. Notably, the accurate evaluation of 3D sketch generation is yet to be resolved due to the absence of ground truth sketches. this work, we measure the CLIP text-image similarity (Radford et al. 2021) (CLIP-Score")) and BLIP-Score (Li al. 2022) metrics to evaluate the consistency of the rendered views with the input text prompt, following previous work (Xing et al. (2024). For all metrics, each view and the input text prompt, and use the averaged value as the final result. Table T shows the evaluation results and method outperforms all the text-to-31) baselines.
render the 3D sketch into 8 views and compute the metric between
8

---

Published as a conference paper at ICLR 2025
Input
CLIPasso
NEF
SDoodle
DifT3DS (Ours)
Figure 6: Qualitative results of the image-to-3D sketch task. 3Doodle and NEF fail to extract reasonable 3D curves from a single input image.
6.3 IMAGE-TO-3D SKETCH
the first to generate 3D sketch from single image. The closest works to ours are Neural Edge Fields (NEF) (Ye al. 2023) and 3Doodle (Choi et al.) 2024), which generate 3D curves from multiview images. We also choose the image-to-2D sketch
Baselines
To the best of our knowledge,
method CLIPasso (iinker et al. 2022) as the additional perceptual reference.
Qualitative Comparisons Fig. 6 shows the qualitative results of the image-to-3D sketch task. Note that CLIPasso can only generate 2D sketch and is used as reference for the results 2D generation methods. As the reconstruction methods, 3Doodle and NEF attain reasonable results for the reference view, but they fail for other views because they lack the ability to predict the unobserved regions from single input image. In contrast, method generate view-consistent 3D sketches.
Quantitative Comparisons We collect 25 images generated with Imagine (Ima 2023 Follow previous works (Qian et al.) 2024, Choi et al. 2024), we use CLIP visual similarity (Radford et al.) 2021) (CLIP-Score') metric to measure the abstract semantic similarity across the reference image and the rendered novel views, while employing LPIPS (Zhang et al. 2018) metric in the reference view to measure structural semantic similarity. For the CLIP-Score', we calculate the metric between each of the 8 rendered images and the reference image, and use the average value the final score. Table reports the evaluation results. On the LPIPS metric, our method significantly outperforms NEF and achieves score comparable 3Doodle, which has optimized for the reference view with more losses, including the LPIPS and CLIP loss. On the CL.PP-core metric, our method surpasses all baseline approaches. These results demonstrate the superiority of method.
User Study We further conduct a user study to evaluate the overall quality of our image-to- 3D sketch results. Specifically, randomly-ordered 3D sketches generated using three methods: NEF, 3Doodle and Diff3DS. We used the Likert scores as the evaluation metric, with a range from to 5 where a higher score indicates that the generated 3D sketch
prepared 25 tasks, each of which is composed of three
preferred by the users. In each task, the participants were asked to score each 3D sketch result based on the following two questions: (1) How well does the 3D sketch fit the input image, c.g., in terms of keeping the similar shape and structure of the input? (2) How is the quality of the 3D sketch, e.g., whether the 4 rendered views are 3D consistent or whether the sketch is visually pleasing? We distributed questionnaires to 40 participants who are CS, EE and Math students and researchers, and got 1000 valid scores in total. The results are summarized in Fig. 7 which highlights the superiority of our method. More details can be found in the Appendix
1.78
2.12
4.18
3Doodle
DifB3D
Figure 7: Bar plots of study results. The score is of scale 1-5, the higher the better.

---

Published as a conference paper at ICLR 2025
6.4 ABLATIONS AND ANALYSIS
Effect of Hyperparameters Fig. 8 illustrates the effect of hyperparameters. Fig. (a) illustrates the effect of different CFG weights À in the text-to-3D sketch task. A larger weight leads to stronger shape prior, while a smaller weight results in shape degradation. Fig. g (b) illustrates the effect of various pre-trained models. The Janus Problem easily arises when using Stable Diffusion, and MVDream partially mitigates this issue (e.g.. the bicycle with three wheels in the bird's-eye view). Then Zero-I-to6 is sufficiently 3D consistent in certain examples (e.g., the upper part of the scissors is bent in the side view). Fig. & (c) illustrates that increasing the initial curve number can enhance the details of results, even though the final curve number is automatically determined by Dynamic Noise Deletion. Too few curves may fail to represent complete 3D object.
Based Stable Diffusion
Based Zero-1 1-to-1
26
112 N..
Am 50
Based MVDream
Based Stable Zeroi22
X 50
(a) CFG Weight.
(b) Pre-trained Models.
(c) Initial Curve Number.
Figure 8: Effect of hyperparameters.
Analysis of Designed Rasterizer Our rasterizer accurately renders the blended colors of overlapping curves and faithfully reproduces occlusion re- lationships between curves, and en- ables seamless integration with other 3D sketch generation methods like 3Doodle. assess effectiveness, we conduct an experiment to generate colored 3D sketches from multi-view images using both the original 3Doodle and variant integrated with rasterizer. As shown in Fig. 9 the original 3Doodle suffers from noticeable color conflict errors across multiple viewpoints (e.g., the support base, which is farther from the camera, is incorrectly rendered in the foreground, occluding the LEGO excavator). These noticeable color conflicts disrupt the object's 's visual coherence and lead to semantic ambiguity. In contrast, our rasterizer consistently reproduces accurate occlusion relationships between objects, providing clearer visual and spatial semantics. Additional results and details can be found in the Appendix D.8
3Doodle
Target Object
3Doodle (+ Our rasterizer)
Figure 9: Analysis of designed rasterizer.
CONCLUSIONS
In this paper, we propose Diff3DS, novel differentiable rendering framework for generating view- consistent 3D sketch by optimizing 3D parametric curves under various supervisions. By employing the recent Score Distillation Sampling (SDS) to distill prior knowledge from pre-trained 2D image generation model, we achieve 3D sketch generation from the flexible text or single image input. Our proposed rasterizer accurately renders the blended colors in overlapping regions and faithfully reproduces occlusion relationships between curves, showcasing its strong potential for generating colored 3D sketches. One limitation of Diff3DS is it inherits the sparse gradient issue from DifTVG and can only optimize continuous parameters. Also, the initial curve number is set manually to achieve balance between the approximation accuracy and complexity of the results. Moreover, our current implementation does not distinguish between the view-independent curve (e.g., feature lines) and the view-dependent curve (e.g., contours of smooth surface boundaries), limiting the expressive capacity of the overall shape of a 3D object, which is extensively discussed in 3Doodle. Incorporating the diverse curve representations of 3Doddle to Diff3DS a promising future work. Also, one future direction is to extend our framework for scene-level generation task, such as text-to-3D scene sketch.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
This work was supported in part by National Natural Science Foundation of China (62202199).
REFERENCES
Random-access renderiin of general vector graphics. URL wtt.:ppp/tx.......//////////////
I409060.1409088
Stable
diffusion
2.1.
attp::/hagglloffac.coost11111111
=rable-diffusion-2-
Imagine. Itttp:://magiing metaa.commme.commm 2023.
Stable zero123.
2023.
Rahul Arora and Karan Singh. Mid-air drawing of curves on 3d surfaces in virtual reality, ACM Transactions on Graphics, 2021.
Seok-Hyung Bae, Ravin Balakrishnan, and Karan Singh. Ilovesketch: cettuuaalaa-possiiiie sketching system for creating 3d curve models. In Proceedings of the Annual ACM Symposium on User Interface Software and Technology, 2008.
Ayan Banerjee, Nityanand Mathur, Josep Lladós, Umapada Pal, and Anjan Dutta. Svgcraft: Be- yond single object text-to-svg synthesis with comprehensive canvas layout. arXiv preprint arXiv:2404.0000 004 /2, 2024.
Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3d: Disentangling geometry and appear- ance for high-quality text-to-3d content creation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023.
Changwoon Choi, Jaeah Lee, Jaesik Park, and Young Min Kim. 3doodle: Compact abstraction of objects with 3d strokes. ACM Trans. Graph., 2024.
Blender Online Community, Blender - 3D modelling and rendering package. Blender Foundation, 2018. URL htt:::/wwww.bleeder
Kevin Frans, Lisa Soros, and Olaf Witkowski. Clipdraw: Exploring text-to-drawing synthesis through language-image encoders. Advances in Neural Information Processing Systems, 2022.
Francisco Ganacim, Rodolfo S. Lima, Luiz Henrique de Figueiredo, and Diego Nehab. Massively- parallel vector graphics. ACM Transactions on Graphics, 2014.
Cindy Grimm and Pushkar Joshi. Just drawit: 3d sketching system. In Proceedings of the International Symposium on Sketch-Based Interfaces and Modeling, 2012.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurlPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
Yukun Huang, Jianan Wang, Yukai Shi, Boshi Tang, Xianbiao Qi, and Lei Zhang. Dreamtime: An improved optimization strategy for diffusion- 3d generation. In Proceedings of the International Conference on Learning Representations, 2024.
Ajay Jain, Ben Mildenhall, Jonathan T Barron, Pieter Abbeel, and Ben Poole. Zero-shot text- exxt-guided object generation with dream fields. In Proceedings of the IEEEJCVF Conference on Computer Vision and Pattern Recognition, 2022.
Ajay Jain, Amber Xie, and Pieter Abbeel. Vectorfusion: Text-to-svg by abstracting pixel-based diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
Ying Jiang, Congyi Zhang, Hongbo Fu, Alberto Cannavò, Fabrizio Lamberti, Henry Y K Lau, and Wenping Wang. Handpainter - 3d sketching in vr with hand-base physical proxy. In Proceedings of the ACM CHI Conference Human Factors in Computing Systems, 2021.
11

---

Published as a conference paper at ICLR 2025
Yongkwan Kim and Beok-Hyung Bae. Sketchingwithhands: 3d sketching handheld products with first-person hand posture. In Proceedings of the ACM Symposium on User Interface Software and Technology, 2016.
Yongkwan Kim, Kyuhyoung Hong, and Junwon Yang. Feather: 3d sketchbook light as feather. In ACM SIGGRAPH Appy Hour, SIGGRAPH '23, 2023.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified visoonlanguage understanding and generation. In Proceedings of the International Conference Machine Learning, 2022.
Tzu-Mao Li, Michal Lukáč, Michaël Gharbi, and Jonathan Ragan-Kelley, Differentiable vector
graphics rasterization for editing and learning, ACM Transactions on Graphics, 2020.
Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming- Yu Liu, and Tsung-Yi Yi Lin. Magic3d: High-resolution text-to-3d content creation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
Lingjie Liu, Duygu Ceylan, Cheng Lin, Wenping Wang, and Niloy J Mitra. Image- based reconstruc- tion of wire art. ACM Transactions on Graphics, 2017.
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-I-to-3: Zero-shot one image to 3d object. In Proceedings of the IEEE/CVF International Conference Computer Vision, 2023.
David Loffredo. Fundamentals of step implementation. STEP Tools, Inc, 1999.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In Proceedings of the European Conference on Computer Vision, 2021.
Nasir Mohammad Khalid, Tianhao Xie, Eugene Belilovsky, and Tiberiu Popa. Clip-mesh: Gener- ating textured meshes from text using pretrained image-text models. In ACM SIGGRAPH Asia Conference Proceedings, 2022.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems, 2019.
Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. Proceedings of the International Conference on Learning Representations, 2023.
Thomas Porter and Tom Duff. Compositing digital images. In Proceedings of the Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH '84, 1984.
Guocheng Qian, Jinjie Mai, Abdullah Hamdi, Jian Ren, Aliaksandr Siarohin, Bing Li, Hsin- Ying Lee, Ivan Skorokhodov, Peter Wonka, Sergey Tulyakov, and Bernard Ghanem. Magic123, One image to high-quality 3d object generation using both 2d and 3d diffusion priors. In Proceedings of the International Conference Learning Representations, 2024.
Zhiyu Qu, Tao Xiang, and Yi-Zhe Song. Sketchdreamer: Interactive text-augmented creative sketch ideation. In Proceedings of the British Machine Vision Conference, 2023.
Zhiyu Qu, Lan Yang, Honggang Zhang, Tao Xiang, Kaiyue Pang, and Yi-Zhe Song. Wired perspec- tives: Multi-view wire art embraces generative ai. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In Proceedings of the International Conference on Machine Learning, 2021.
12

---

Published as a conference paper at ICLR 2025
Peter Schaldenbrand, Zhixuan Liu, and Jean Oh. Styleclipdraw: Coupling content and style in text-to-drawing translation. In Proceedings of the International Joint Conference on Artificial Intelligence, 2022.
Thomas W Sederberg and Geng-Zhe Chang. Isolator polynomials. In Algebraic Geometry and Its Applications, 1994.
Yichun Shi, Peng Wang, Jianglong Ye, Long Mai, Kejie Li, and Xiao Yang. MVDream: Multi- view diffusion for 3d generation. In Proceedings of the International Conference on Learning Representations, 2024.
Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation, In Proceedings of the International Conference on Learming Representations, 2024.
Vikas Thamizharasan, Difan Liu, Shantanu Agarwal, Matthew Fisher, Michaäl Gharbi, Oliver Wang, Alec Jacobson, and Evangelos Kalogerakis. Vecfusion: Vector font generation with diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024a.
Vikas Thamizharasan, Difan Liu, Matthew Fisher, Nanxuan Zhao, Evangelos Kalogerakis, and Michal Lukac. Nivel: Neural implicit vector layers for text-to-vector generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024b.
Kenji Tojo, Ariel Shamir, Bernd Bickel, and Nobuyuki Umetani. Fabricable 3d wire art In ACM SIGGRAPH Conference Proceedings, SIGGRAPH '24, 2024.
Christina Tsalicoglou, Fabian Manhardt, Alessio Tonioni, Michael Niemeyer, and Federico Tombari. Textmesh: Generation of realistic 3d meshes from text prompts. In Proceedings of the International conference on 3D vision, 2024.
Yael Vinker, Ehsan Pajouheshgar, Jessica Bo, Roman Christian Bachmann, Amit Haim Bermano, Daniel Cohen-Or, Amir Zamir, and Ariel Shamir. Clipasso: Semantically object sketching. ACM Transactions on Graphics, 2022.
Yael Vinker, Yuval Alaluf, Daniel Cohen- and Ariel Shamir. Clipascene: Scene sketching with different types and levels of abstraction. In Proceedings the IEEE/CVF International Conference Computer Vision, 2023.
Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A Yeh, and Greg Shakhnarovich. Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023a.
Peng Wang and Yichun Shi. Imagedream: Image-prompt multi-view diffusion for 3d generation. arXiv preprint urXiv:2312.0222 2023.
Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolific- dreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. In Advances in Neural Information Processing Systems, 2023b.
Gerold Wesche and Hans-Peter Seidel. Freedrawer: free-form sketching system on the responsive workbench. In Proceedings of the ACM Symposium on Virtual Reality Software and Technology. 2001.
William T. Vetterling William H. Press, Saul A. Teukolsky and Brian P. Flannery. Numerical recipes 3rd edition: The art of scientific computing (3 ed.). cambridge, 2007.
Markus Worchel and Marc Alexa. Differentiable rendering of parametric geometry. ACM Transactions on Graphics, 2023.
Tong Wu. Zhibing Li, Shuai Yang, Pan Zhang, Xingang Pan, Jiaqi Wang, Dahua Lin, and Ziwei Liu. Hyperdreamer: Hyper-realistic 3d content generation and editing from a single image. In ACM SIGGRAPH Asia Conference Proceedings, 2023.
13

---

Published as a conference paper at ICLR 2025
Min Xin, Ehud Sharlin, and Mario Costa Sousa. Napkin sketch: handheld mixed reality 3d sketching. In Proceedings of the ACM Symposium on Virtual Reality Software and Technology. 2008.
XiMing Xing, Chuang Wang, Haitao Zhou, Jing Zhang, Qian Yu, and Dong Xu. Diffsketcher: Text guided vector sketch synthesis through latent diffusion models. In Advances in Neural Information Processing Systems, 2023.
Ximing Xing, Haitao Zhou, Chuang Wang, Jing Zhang, Dong Xu, and Qian Yu. Svgdreamer: Text guided svg generation with diffusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
Zhijin Yang, Pengfei Xu, Hongbo Fu, and Hui Huang. Wireroom: model-guided explorative design of abstract wire art. ACM Transactions on Graphics, 2021.
Yunfan Ye, Renjiao Yi, Zhirui Gao, Chenyang Zhu, Zhiping Cai, and Kai Xu. Nef: Neural edge fields for 3d parametric curve reconstruction from multi-view images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
Taoran Yi, Jiemin Fang, Junjie Wang, Guanjun Wu, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Qi Tian, and Xinggang Wang. Gaussiandreamer: Fast generation from text to 3d gaussians by bridging 2d and 3d diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2024.
Emilie Yu, Rahul Arora, Tibor Stanko, J. Andreas Bærentzen, Karan Singh, and Adrien Bousseau. Cassie: Curve and surface sketching in immersive environments. In Proceedings of the ACM CHI Conference on Human Factors in Computing Systems, 2021a.
Xue Yu, Stephen DiVerdi, Akshay Sharma, and Yotam Gingold. Scaffoldsketch: Accurate industrial design drawing in vr. In Proceedings of the ACM Symposium on User Interface Software and Technology, 2021b.
Ya- Ta-g Yue, Xiaolong Zhang, Yongliang Yang, Gang Ren, Yi-King Choi, and Wenping Wang. Wiredraw: 3d wire sculpturing guided with mixed reality. In Proceedings of the ACM CHI Conference Human Factors in Computing Systems, 2017
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018.
Junzhe Zhu, Peiye Zhuang, and Sanmi Koyejo. HIFA: High-fidelity text-to-3d generation with advanced diffusion guidance. In Proceedings of the International Conference on Learning Repre- sentations, 2024.
14

---

Published as a conference paper at ICLR 2025
SUPPLEMENTARY
OVERVIEW
This supplementary material is structured into multiple sections that offer further details and analysis pertaining to our Dift3DS research. Concretely, will hone in on the topics that follow:
In section
provide the implementation details of Diff3DS.
In section
we provide a detailed proof for the 3D rational Bézier curve projection.
In section we provide detailed analysis for finding the closest distance between a point
and rational Bézier curve.
In section
we provide more analysis and experiment results.
In section we provide more high quality sketch results.
A IMPLEMENTATION DETAILS
We implement rendering framework in C++/CUDA with the PyTorch interface, and follow the STEP standard (ISO 10303-21) (Loffredo 1999) to format and save our 3D curve results. In the experiment, user-specified number of curves will be randomly initialized within sphere of radius 1.5. The default curve number is set to 56. We randomly sample the camera position using the radius from 1.8 2.0, with the azimuth in the range of -18 to 180 degrees, the elevation in the range of to 30 degrees and the field of view (fov) of 60 degrees. Notice that Stable-Zero 123 abnegates the relative distance for simplifying( we its radius 2.0.
For the mre-trained model, we apply Stable Diffusion 2.1 (sta) and MVDream (Shi et al. (024)) for the text-to-3D sketch task, with the CFG weight of 50. And we apply Zero-I-to-3 (Liu et (022)) and Stable-Zero 23 Sta 2023) for the image-to-3D sketch task, with the CFG weight of 7.5 followed by
(Liu et al. 2023 Sta 2023)
For all tasks, the total number of training steps is 4000. Starting from step 2000, we dynamically delete the noise every 100 steps. The training process requires hour for the text-to-3D sketch task and 2 hours for the image-to-3D sketch task a single NVIDIA A10 GPU with batch size of 4. To optimize the control point positions, we use the Adam optimizer and set the learning rate of the optimizer to 0.002. For the time annealing schedule, prefer to decrease the maximum and minimum time steps from 0.85 to 0.3 and 0.1, respectively, over the first 3600 steps. Due to limited resources, not all possible combinations of the fully explored, and there may be other configurations that could produce hetter results.
related to the time steps have been
For the Dynamic Noise Deletion, only the curves with lengths below specified threshold will be considered as noise and removed dynamically. The trade-off between noise removal and detail preservation can be balanced by adjusting the threshold. Currently, the threshold empirically 0.1 which is below the 10% of the average curve length.
B 3D RATIONAL BÉZIER CURVE PROJECTION
Given the control points P; = (Prr pa P3) E R², the associated non-negative weights wy and the
Bemstein polynomials Bi (f), for 0 < define nth-degree 3D rational Bézier curve p(t) by:
10
(1)
p(t) Σ
P (a(t), (()),(()))
Byn(t
Assume the focal plane is z = f, where f is the focal length, the corresponding projected 2D curve p(t) under the pinhole camera perspective projection can be defined as:
z(1) 7(1)
p()) = (z((), (())
3(t) 2(t) n.
15

---

Published as a conference paper at ICLR 2025
Due to rational Bézier curves projective invariant, the projected 2D curve ptt) still is identical to the rational Bézier curve defined by projected control points. Given projected 2D control points
P.
(
p(t) can be written as 2D rational Bézier curve with adjusted weights uy ü; P.::
z(t) f, j(t)) 3(1) 2(1)
p(t)
Ei (L)
E Bi (1), PM f)
S
By,n(t)ū (t)ū
Σj= (t,n(t
Ban (t;
E
(U
p p 1
oi= Byn (w)
P.
Banlt
Bi (1);;
n
Baa (1);; ppp 2== (t)n(
p.: Σ
اド=D Bj,n(t)u) (t)w
i=0
((())((())
B (D)w P.. Σラ= (t)
C
CLOSEST DISTANCE BETWEEN A POINT AND A RATIONAL BÉZIER CURVE
Given point E I22 and neth-degree rational Bézier curve
p(t) Ei= (t) Ey. (t)
we want to find the closest distance between them and the closest point / on the curve p. The closest distance is defined (p(t*) 9)² and 1 is defined as:
arg, min (p(t) 2
To solve the t², we take the derivative of the squared distance with respect to and set zero:
2(p(t) q)p (t) p(t) O,
which equal to solve the roots of a polynomial with degree 3n 2. For the quadratic rational curve and the cubic rational curve, a 4th order polynomial and 7th order polynomial need to be solved respectively. We provide the detail separately in section C.I and section C.2
To solve all real roots of these polynomials, inspired by DiffVG, we solve the polynomial using bisection and the Newton Raptson method (William H. Press & Flannery, 2007). The Newton- Raphson solver obtains its initial guess for intervals from isolator polynomials (Sederberg & Chang. 1994). For the polynomial p(t) with two adjacent real roots t and tz. given any two lower order other polynomials b(t) and c(t), define
a(t) = b(t)p' (t) + c(t)p(t),
where p' (t(t) is the derivative, (Sederberg & Chang 1994 proofs that a(t) or b(t) must has least one real root in the closed interval (Yo 41 Since p(ln)
p(h) O, we know a(lo ((6))a(4)) =
b(to)((())) pl (11). Since to and L are roots of p and of (L) S O. Thus either ((l))a((1) S
0 or b(to ((t1)) 0. Please see the original paper for discussions multiple roots.
After solving all the real roots of and within the (0, 1), the intervals for finding the real roots of p are then determined. We can then use bisection and the newton-Raphson method to find all the
roots of p within these intervals. The key operation is the selection of the polynomials b(t) and c(t). (Sederberg & Chang. 1994) prefer to find the b(t) and c(t) which follow Degree(a) Degree(b) Degree(p) 1. In the upcoming paragraphs, we will discuss the cases of the 4th order polynomial and the 7th order polynomial in section C.3 and section C.4
16

---

Published as a conference paper at ICLR 2025
C.1 CLOSEST DISTANCE OF QUADRATIC RATIONAL CURVE CASE
The quadratic rational curve p()) can be reorganized using variables A through F as follows:
p(() (1 - w) PoPo Po + 2(1 1)te, + &o222
Al + BL+ + C Dt2 Et + F
(1 u) 2(1 t)tua f2u02
can further calculate p(t) 9 and pt) as:
p(1) q = (A Dq)t² +(B Eq)t + (C- Fq)
Dt. + Et + F
((()) (AE BD + (2AF -2CD)t -BF-CE) CE)
(D12 Ft + F)2
By reorganizing them using variables a through f:
at + bt + c
p(t) 9 Dt2 Et + F
dt? + et +
(1()) = (D12 + El + F)2²
we can rewrite the target polynomial p(t) as below:
p(t) 2(p(t) (t) (at + bt + (dt + et + f)
(DP + Et F)3
whose numerator is a 4th order polynomial in the variable Note that the denominator of p(t) is strictly positive, due to the definition of the rational Bézier curve. Therefore, finding the real roots of the equation p(t) = 0 is equivalent to finding the real roots of the 4th order polynomial in numerator.
C.2 CLOSEST DISTANCE OF CUBIC RATIONAL CURVE CASE
The cubic rational curve p(t) can be reorganized using variables A through H as follows:
p(() = (1- u Po + 3(1 tw, P1 + 3(1 - t)(² 22 + Auyys
(1 wo + 3(1 t) + 3(1 t)t² w2 + 1ws
A1 + B1 + Ct. C+++1 D
Ets + F12 + Gt +
we can further calculate p(t) - 9 and pt( as:
(A - tq + (B - Fq)t + (C Gq)t + (D Hq)
p(() 0
E13 + F12 + GL H
3.A² + 2Bt + C
(3Et² +2Ft+ G) Bt + Ct D)
p'(t) Et + F+2 Gt H
(Et Ft2 + Gt Ct++1 H)²
By reorganizing them using variables through i:
at + +
p(t) q -= Et FF + Gt
+t +
v'(1) = (E13 Fi2 Gl + Hy
write the target polynomial p(t) below:
p(t) 2(p(t) - q)p' = (at3 112 ct + (et² f(3 + gt2 + ht + (E+ + Ft² + Gt H)3
whose numerator is a 7th order polynomial in the variable t. Note that the denominator of p(t) is strictly positive, due to the definition of the rational Bézier curve. Therefore, finding the real roots of the equation p(t) = 0 is equivalent to finding the real roots of the 7th order polynomial in the numerator.
17

---

Published as a conference paper at ICLR 2025
C.3 ISOLATOR POLYNOMIALS OF THE 4TH ORDER POLYNOMIAL CASE
We aim to isolate the real roots of the 4th order polynomial p(t) by the roots of two lower order polynomials a(l) and b(l). To keep Degree(a) + Degree(b) Degree(p) we select a(I) as the 2nd order polynomial and b(t) as the linear polynomial.
Define p(t) and (t) as follows:
p()) + + CC +D++E
(t( 4thh 3Bt 2Ct
and we know a(t) c(t)p(t) (t). Let c(t) = att) can be written as a(t) = p
By doing a long division between and p', we obtain the isolator polynomials as follows:
C
3B
3D
CB
DB
a(t)
)t²
) + (c
16
4
16
1
B
b(t) 4 16
+
C.4 ISOLATOR POLYNOMIALS OF THE 7TH ORDER POLYNOMIAL CASE
We aim to isolate the real roots of the 7th order polynomial p(t) by the roots of two lower order polynomials a(t) and b(t). To keep Degree(a) + Degree(b) = Degree(p) we select a(t) and b(t) both be the cubic polynomial.
Define p(t) and (t) as follows:
p(t) = + pot ... +.....10 pit 11 Po
(1()) 7t 6pot +.... + P1:
and we know a(t) c()p(t(())()) b()) We wish a and b to both be cubic polynomials. Note that b(t)p'(t) 9th order polynomial, so we assume e(t) to be a 2nd order polynomial to ensure that c(t)p(t) is 9th order polynomial too. Let b(t) (t + CC + Dt E) and c(t) (7t² + At + B), we can obtain two 9th order polynomials, and their difference a(x) a cubic polynomial:
c(t))(t) = (t" + p&t + + pit (7t + At B) b(t) (7t 6p&t +.... pt) Ct2 Dt + E).
By setting the coefficients of the 8th to 4th order terms of a(t) to O, we can obtain system of linear equations in variables A through E. By solving this system of equations, of variables A through E. We can then proceed to find b(t), c(t), and a(t). Please refer to our code for the specific details.
determine the values
D MORE ANALYSIS AND EXPERIMENTS
D.1 ABLATION STUDY OF DESIGNED COMPONENTS
evaluate the contributions of the Time Annealing Schedule and Dynamic Noise Deletion compo- nents, shown in Fig. 10 The omission of Time Annealing Schedule affects the conformity of the generated results with the semantic meaning of the input text or the level of detail in the reference image, and the omission of Dynamic Noise Deletion leads the retention of some noise in the results.
coleeed growing on log
Input
Our
Wio TAS
Wio DND
Imput
Our
Wio TAS
Wio DND
Figure 10: Ablation study Time Annealing Schedule (TAS) and Dynamic Noise Deletion (DND).
18

---

Published as a conference paper at ICLR 2025
D.2 USER STUDY
We provide details of our user study. Specifically, our user study includes 25 tasks, which cover a diverse range of object types, e.g., flamingo, unicorn, windmill, lighthouse, boat, bike, sword, bow, scissor, flower etc. Each task is composed of three 3D sketches generated using three methods: NEF, 3Doodle and Diff3DS. We used the Likert scores as the evaluation metric, with range from 1 to where a higher score indicates that the generated 3D sketch is more preferred by the users.
In each task, the order of sketches presented the users was randomized. Specifically, for each method, the 3D sketch is rendered in four views and organized in the form shown in Fig. TT Then, for each task, contains three randomly-ordered rows, while each row represents the results of one method. The participants were asked to score each 3D sketch output based on the following two questions:
1. How well does the 3D sketch fit the input image, e.g., in terms of keeping the similar shape and structure of the input?
2. How is the quality the 3D sketch, e.g., whether the 4 rendered views are 3D consistent or whether the sketch is visually pleasing?
We distributed questionnaires to 40 participants who are CS, EE and Math students and researchers, and finally collected total 1000 valid scores.
Irage
Figure 11: Screenshot of example question used in user study
D.3 EFFICIENCY OF THE INITIAL NUMBER OF CURVES
The effect of the initial curve number have reported in Fig. 8 (c). To further evaluate the efficiency of the curve number, we have conducted a new ablation study on the initial curve number, and the results shown in Table 3 The results show that the number of initial curve increases. Ccor-Score performance shows a consistent increase while the average optimization time keeps getting longer. Meanwhile, when the initial number reaches 56, further increasing it 112 does not result in significant performance improvement due to the Dynamic Noise Deletion process.
Metric Number
num=112 12
num=56
num=28
num=14
CLIP-Score" (ViT B/16)
0.3074
0.3046
0.2920
0.2785
CLIP-Score" (ViT B/32)
0.3032
0.3034
0.2883
0.2786
Average Training Time
80min
60min
45min
35min
Table Efficiency of the initial curve number.
19

---

Published as a conference paper at ICLR 2025
D.4 EFFECT OF NOISE SAMPLE SCHEDULE
The choice of noise sampling schedule has significant impact on the results. For our task, the large noise timestep focuses aligning the curves with the semantic content of the input condition during the initial training phase. Conversely, the small noise timestep employed during the later training phase focuses on further enhancing the details. In the current implementation, we have employed a biased sampling that decreases the maximum and minimum time steps from 0.85 to 0.3 and 0.1, whereas original DreamFusion randomly samples noises from 0.98 0.02. Based on experience, further reducing the sampling of high-level noise would lead to a significant decline in the geometric quality of the results. We have provided relevant results in the Fig. 12
~~ (0.300885)
~~ [0.3,
Figure 12: Effect of noise sample schedule.
D.5 EFFECT OF CURVE INITIALIZATION
In the current implementation, the initial curves randomly initialized within sphere of radius 1.5. Based on our observations, the quality of the results is generally insensitive to minor variations the curve initialization radius. However, significantly reducing the radius still degrade the quality of the results, as shown the Fig- [3] We recommend adjusting the radius to ensure the projected curve remains fully visible within the image and occupies reasonable area.
radius
radius 0.5
time
Figure 13: Effect of curve initialization.
D.6 EFFECT OF OPTIMIZING COLOR, OPACITY AND WIDTH
Our proposed differentiable rasterizer supports the optimization of the con- trol point position, color, opacity and curve width. We report the optimiza- tion results in different modes, as shown in Fig. [4] including Position, RGB, RGBA (both color and opacity), Alpha, and Width. Notably, we ob- serve that optimizing the curve width results in unstable outcomes. We hy- pothesize that this instability stems from the rendered image semantics being highly sensitive variations in curve width, and plan to explore this topic in our future work.
Position
RGB
RGBA
Alpha
Width
Figure 14: Effect of optimizing color, opacity and width.
20

---

Published as a conference paper at ICLR 2025
D.7 COMPARISON WITH 3D SKETCH RECONSTRUCTION METHODS UNDER MULTI- VIEW SUPERVISION
We further compare our approach with the 3D sketch reconstruction method under multi-view supervision. Specifically, we employ MVDream and Stable-Zero123 to generate 3D objects from input text and single view, rendering 120 horizontal views as training data. Notably, we report comparisons only between Diff3DS and 3Doodle, as NEF failed to converge and produce reasonable results in our experiments. To ensure a fair comparison, 3Doodle adopts the same setup as our method, optimizing only the 3D curves and randomly initializing the positions of the curve control points. For the text-to-3D and image-to-3l sketch tasks, 3ooddle requires hours (1 hour for object generation) and 2.5 hours (0.5 hours for object generation), respectively. In contrast, method only requires hour and 2 hours, respectively.
As illustrated in Fig. 15 method achieves performance comparable the baseline within more efficient training time.
Mystical elven bow, enchanted arrows
A quill pen, European style, neat feathers
Higly detailed, majestic royal ship
lire phoenix, mythical bird, engulfed in flames
Input
3D Object
3Doodle (120 views, -3h)
Diff3DS (-1h)
(a) Text-to-3D sketch task.
s
Input
3D Object
3Doodle (120 views, ~2.5h)
Diff3DS (-2h)
(b) Image-to-3D sketch task.
Figure 15: Comparison with 3D sketch reconstruction method under multi-view supervision.
21

---

Published as a conference paper at ICLR 2025
D.8 ANALYSIS DESIGNED RASTERIZER
We conduct an experiment generating colored 3D sketches from given multi-view images using both the original 3Doodle and its variant integrated with our rasterizer. For the training dataset, we the "toyhorse" and "toycar" from the 3Deodle-provided dataset, along with "hotdog", "ship". and "lego" from the Nerf Synthesis dataset (Mildenhall al. 202T). As shown in Fig. [6] the original Dooode suffers from noticeable color conflicts across multiple viewpoints (c.g., the plate's s curve incorrectly appears over the hotdog), disrupting the visual coherence of the object and leading semantic ambiguity. In contrast, our rasterizer consistently reproduces accurate occlusion relationships between objects, providing clearer visual and spatial semantics.
3Doodle
3Doodle (+ Our rasterizer)
3Doodle
3Doodle (+ Our rasteriser)
SDoodle
SDoodle Our rasterizer)
3Doodle
Targel Object
3Doodle (+ Our rasterizer)
Figure 16: Analysis of designed rasterizer. the figure, the red arrow points to the curves which has incorrect depth order, c.g., the curve of the plate falsely appears over the hotdog.
D.9 SKETCH STYLIZATION
The style of our generated sketches can be altered by applying different brushes to the vector strokes. We rendered the 3D sketch in the vector graphics format and used various brush styles from Adobe Illustrator to demonstrate the diverse styles of the generated sketches, as shown in Fig. 17
Figure 17: Sketch stylization.
22

---

Published as a conference paper at ICLR 2025
E MORE HIGH QUALITY RESULTS
E. MORE RESULTS OF THE TEXT-TO-3D SKETCH TASK
"Mystical elven bow, ethereal craftsmanship, enchanted arrows, forest protector."
"A DSLR photo of a time clock, clear pointer."
"A flower office table, solid wood, minimalist, straight table legs." "
"A quill pen, vintage, European style, three-dimensional, neat feathers."
"Castle in the clouds, ethereal fortress, sky-high citadel."
Figure 18: More results of the text-to-3D sketch task.
23

---

Published as a conference paper at ICLR 2025
E.2 MORE RESULTS OF THE IMAGE-TO-3D SKETCH TASK
Input
Generated Results
Figure 19: More results of the image-to-3D sketch task
24

---

Published as a conference paper at ICLR 2025
E.3 MORE 3D SKETCH RESULTS RENDERED USING BLENDER COMMUNITY (0018)
Figure 20: More rendered 3D sketch results.
25

---

Published as a conference paper at ICLR 2025
Figure 21: More rendered 3D sketch results.
26


Reviews:
[{"review_id": "mqI896OIbm", "reviewer": "Reviewer_w44R", "summary": "The authors present Diff3DS, a framework for optimizing 3D strokes using text or image guidance. This work introduces a novel extension of DiffVG that allows users to differentiably rasterize 2D projections of 3D parametric curves with more accurate depths / occlusions based on the 3D location of the curves. Using this new rasterization technique, Diff3DS applies SDS supervision to generate view-consistent 3D sketches using text or image guidance. This work validates its approach through ablations of key components and comparisons to other similar works. Diff3DS shows improved performance over other methods in both automated metrics and a perceptual user study. The paper is well written and easy to follow. The method is explained in sufficient detail and design decisions are properly motivated. Informative figures clearly showcase the contributions of this work (i.e. Figure 2 nicely demonstrates this ability to have non-uniform ordering between curves). The provided experiments validate claims made in the paper.", "questions": "It appears from Figure 6 that, while your approach produces more 3D consistent results than 3Doodle, it less closely captures the structure of the guiding image. Do you have a sense of why that might be? Is it due to the fact that Diff3DS is more 3D consistent and thus is less able to overfit to exact structural details? Can this be improved?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Strengths:\n- Through the use of score distillation sampling for supervision, this work is the first to enable text/image driven 3D stroke generation from scratch (as opposed to existing works which simplify / abstract / sketchify an existing 3D representation defined by multi-veiw images).\n- The novel differentiable rasterization technique introduced by this work allows for more accurate depths when rasterizing 3D curves.\n- The paper is well written and organized.", "weaknesses": "Weaknesses:\n- The main contribution of this work appears to be the improved differentiable rasterization approach that enables more accurate depths during rasterization. However, it is not clear that this component is responsible for improved performance. Figure 11 ablates the impact of this rasterization technique on 3Doodle. However, the differences are quite minor and at this level of abstraction, the depths of these curves seem less important to the visual coherence of the object.\n- The comparisons currently shown do not use strong enough / similar enough baselines. Some of the comparisons shown are to standard (non-sketch) generative approaches. Showing these is fine, especially given the lack of existing work on text driven 3D consistent sketch generation. However, it feels that using score distillation guidance + DiffVG should still produce strong results for text/image guided curve generation. This is an important baseline and should be compared to in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rPZHzVrT0b", "reviewer": "Reviewer_Jp4d", "summary": "The paper proposes a framework for generating 3D sketches from either a text prompt or a single image. It is based on differentiable rendering of 3D Bézier curves and applies supervision from pretrained image diffusion models from random viewpoints.", "questions": "* Why care about z-ordering of the curves? In which situation is it relevant?\n* Figure 4: The meaning is unclear. Is this supposed to represent a pseudo-ground truth image? How does it relate to gradient supervision?\n* In which situation are 3D sketches useful?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* Covers multiple aspects of differentiable rendering.\n* Extends DiffVG, a differentiable rasterizer for 2D vector graphics, to 3D Bézier curves.\n* Presents a relatively simple and elegant pipeline, clearly summarized in Figure 3.\n* Includes small but useful techniques that are well explained and analyzed:\n    * Time annealing for the Score Distillation Sampling (SDS).\n    * Curve filtering to reduce noise.", "weaknesses": "* The z-ordering of Bézier curves in Section 4.3 is the most technically advanced contribution. But the motivation is unclear: all sketches consist of black strokes, and intersections are rare at the image scale. Why care about correctly ordering the curves? Figure 11’s ablation shows only very minor visual differences. The effect of this ordering on 3D sketch generation is also left unclear.\n* In the related works section, the motivations against existing 3Doodle approaches seem weak:\n    * Issues with perspective projections.\n    * Use of z-ordering for handling colored intersections.\n* The usefulness and relevance of 3D sketches as shape representation is not fully clarified.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "I00JzDDGb1", "reviewer": "Reviewer_cfcy", "summary": "Nowadays, there are a lot of works conducting optimization of 3D models from 2D diffusion priors with SDS loss. Insteading of optimizing 3D models, this work aims to output 3D sketch which is also an important 3D format in related areas. To do so, a novel differatiable rendering framework is proposed. Experimental results validate the effectiveness of the proposed methods. Compared with existing differenatiable curve rendering methods, such as 3Doodle, the proposed method can maintain depth ordering which is very helpful for the opimization.", "questions": "No more questions.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- In general, the paper is very well-presented. \n- To make 3D sketch rendering differienable is important and challenge. The paper proposed a novel one which enables the maintaining of the depth ordering. I also believe such rendering framework could benefit some other tasks. \n- The results seem very good. It's far better than existing ones visually.", "weaknesses": "There are also some issues which can be improved. \n1) Disscussions on the practical usage of 3D sketches can be improved. After the first glance of this paper, my primary curiousity is from the usage of 3D sketches. In the paper, the authors just mentioned \"widely used\" but did not provide examples. For me, it is really hard to imagine the usage of such 3D format. When thinking this in deep, I think 3D sketch/line can be an abstract representation of 3D structure, which can be an intermediate representation for some tasks like 3D reconstruction. In this senario, the differentiable rendering will also be very important. More discussions on this are strongly encouraged to make the application senarios more clear. More specifically, I think a subsection is needed to discuss those applications of 3D sketches. \n\n2) The evaluation is somewhat weak. As mentioned in the title and intro, view-consistency is the target of the proposed method. Howerver, there are only one user study to measure the view-consistency which is weak. The authors is encouraged to present some numeric results of view-consistency. For example, given two adjacent views, we can do a warping based on the optical flow estimation and then calculate the alignment error. If two views are consistent, the error should be minor. This metric can be compared with some baselines such as zero-123. \n\n3) I also have a concern about the number of curves. It seems the number of curves is a parameter needs to be set manually. This will increase the burden of the users. If it is so, please include this discussion into the limitation part.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wl5iQo6QFq", "reviewer": "Reviewer_TMKm", "summary": "The paper introduces Diff3DS, a novel approach for generating view-consistent 3D sketches from either text prompts or image inputs. It employs a differentiable rendering framework that enables the optimization of 3D curve parameters through fully differentiable rasterized images. The method first projects parametric 3D curve control points into 2D using perspective projection. It then applies differentiable rasterization, similar to DiffVG, while correctly handling depth during curve intersections. The framework leverages Score Distillation Sampling (SDS) for supervision, using either a pretrained text-to-image model or an image-guided model. Additional techniques, such as dynamic noise deletion and time annealing schedules, further enhance optimization and generation quality.\nExperiments show that Diff3DS generates consistent 3D sketches from both text and image inputs, outperforming previous sketch synthesis methods in terms of CLIP visual similarity and user preference.", "questions": "1. In some examples, such as the phoenix, the 3D curves appear insufficient to fully capture the object's shape, and the curves still seem somewhat disorganized. Can this be solved with a higher number of 3D curves?\n\n2. The method initializes curves within a fixed-size sphere. How heavily does this initialization impact the final optimization?\n\n3. Why not consider optimizing additional parameters, such as the width, color, or transparency of the 3D curves? Could optimizing these properties lead to significantly better results?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper introduces a novel framework for 3D sketch generation, which, to the best of my knowledge, is the first to support generation from both images and text prompts.\n\n2. The proposed method employs differentiable rendering that supports perspective projection and occlusion-aware rasterization for 3D curves, distinguishing it from earlier approaches.\n\n3. Experimental results demonstrate that Diff3DS outperforms prior methods in both semantic alignment and user-preferred quality.\n\n4. The writing is clear and the structure of the paper is easy to follow.", "weaknesses": "Although the paper is well-written and the generated sketches align with the given image and text inputs, there are concerns about the sufficiency of the contribution and the depth of the experimental evaluation:\n\n1. One of the claimed novelty is the use of perspective projection with depth-aware rasterization. However, it seems this approach is not entirely new. A similar perspective projection has already been discussed in 3Doodle (specifically in Appendix A.2), which seems to have no clear difference with the procedure proposed in this paper.\n\n2. While fixing occlusions in 3D curves is a positive addition to the rasterizer, there is no conclusive experimental evidence showing how this improvement impacts the quality of generation. For instance, it remains unclear whether depth-aware rasterization significantly enhances the final results beyond the occlusion analysis in Figure 11.\n\n3. Another novelty is the use of SDS loss supervision, which has not been employed by previous methods. However, applying SDS loss to rasterized images is relatively straightforward and does not present a significant technical difficulty, given its widespread use in NeRF-related tasks. This loss could be easily incorporated into prior approaches without altering their core components. Moreover, the experiments do not convincingly demonstrate the advantage of direct SDS supervision on 3D curves, which I elaborate below.\n\n4. In the image-to-3D sketch task, Diff3DS is compared with 3Doodle and NEF, showing better CLIP scores and improved 3D consistency. However, the fairness of this comparison is questionable, as Diff3DS benefits from multi-view supervision via SDS loss, while 3Doodle and NEF are trained only from a single reference view. Naturally, multi-view supervision leads to better 3D consistency. I think a better comparison would be another baseline like this: first train a 3D representation such as NeRF using the text2image or image2image SDS loss, then use the multi-view renderings of this representation to provide multi-view supervision for training other 3D sketch reconstruction methods, followed by a comparison in terms of quality and efficiency to truly demonstrate the benefits of directly optimizing 3D curves with SDS loss.\n\nGiven above points, I believe the paper's contributions may not be substantial enough to justify its acceptance. However, I am open to raise my score if the authors can provide a convincing explaination to the above concerns.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)