Diff3DS: Generating View-Consistent 3D Sketch via Differentiable Curve Rendering

## Abstract

Abstract 3D sketches are widely used for visually representing the 3D shape and structure of objects or scenes.
However, the creation of 3D sketch often requires users to possess professional artistic skills.
Existing research efforts primarily focus on enhancing the ability of interactive sketch generation in 3D virtual systems.
In this work, we propose Diff3DS , a novel differentiable rendering framework for generating view-consistent 3D sketch by optimizing 3D parametric curves under various supervisions.
Specifically, we perform perspective projection to render the 3D rational Bézier curves into 2D curves, which are subsequently converted to a 2D raster image via our customized differentiable rasterizer.
Our framework bridges the domains of 3D sketch and raster image, achieving end-to-end optimization of 3D sketch through gradients computed in the 2D image domain.
Our Diff3DS can enable a series of novel 3D sketch generation tasks, including text-to-3D sketch and image-to-3D sketch, supported by the popular distillation-based supervision, such as Score Distillation Sampling (SDS).
Extensive experiments have yielded promising results and demonstrated the potential of our framework.
Project: https://yiboz2001.github.io/Diff3DS/

## 1 Introduction

3D sketches, which utilize strokes to emphasize abstraction and visually encapsulate the 3D shape and structure of objects or scenes, serve as indispensable tools for visualizing concepts and ideas.
Existing research efforts have primarily focused on exploring how to enable interactive creation of 3D sketch in 3D virtual systems [, , ] and immersive environments [, , , , , , , ].
Such interactive 3D sketch creation tools provide artists with unprecedented freedom, allowing them to draw their inspirations directly in the immersive 3D environment.
However, it usually requires users to possess professional artistic skills and experience for 3D operation, making the 3D sketch creation less friendly for ordinary users.
There is a notable absence of a user-friendly 3D sketch generation method in the community.

In recent years, content reconstruction or generation via differentiable rendering has gained great attention in the fields of computer graphics and vision research.
The goal is to convert the traditional rendering pipeline into a differentiable image synthesis process and employ the 2D image supervision to optimize the object or scene representation.
For example, NeRF [] bridges the 3D neural representation and raster image domain with a differentiable volume rendering pipeline, and has contributed to the further development of the 3D generation [, , , , ] by leveraging the continuous advancements in multimodal supervision [, ].
For the generation of vectorized or parametric content, differentiable rendering has also been investigated [, , , , , , , , , , ].
One representative work is DiffVG [], which proposes a differentiable 2D vector graphics rasterizer to compute gradients from raster images and optimize the vectorized image (e.g., 2D Bézier curves and other parametric shapes) via back-propagation.
For generation of sketch, CLIPasso generates 2D sketches using a differentiable rasterizer to optimize the parameters of the Bézier curves directly with respect to a CLIP-based perceptual loss.
Recently, 3Doodle [] generates view-consistent 3D sketches of the target object by directly optimizing the parameters of 3D strokes to minimize the perceptual losses for given multi-view images.

Although 3Doodle achieves promising results for generating 3D sketches, it has two main limitations.
First, 3Doodle performs orthogonal projection to obtain the 2D sketch rendering and ignores the depth order of the curves, which may lead to errors in the 3D sketch.
Second, its optimization requires supervision from multiview images, which makes it more like a reconstruction pipeline instead of a more flexible generation pipeline.
In this paper, we aim to solving above issues and present Diff3DS, a novel differentiable rendering framework for generating view-consistent 3D sketch from flexible inputs such as a single image or text.
Specifically, we represent the 3D sketch as a set of 3D rational Bézier curves and perform the perspective projection to obtain the 2D rational Bézier curves.
Then, we propose a new differentiable rasterizer based on DiffVG [] to accurately render the projected 2D curves based on the depth order.
Our framework supports end-to-end optimization of 3D curve primitives under flexible supervisions such as the distillation-based loss.
By employing the recent Score Distillation Sampling (SDS) [, ] to distill prior knowledge from pre-trained 2D image generation model, we achieve 3D sketch generation from the text or single image input.
We conduct quantitative and qualitative comparisons between Diff3DS and related methods, and the results show the superiority of our approach in generating view-consistent 3D sketch.
Ablation studies also verify the effectiveness our our key hyperparameters and components.

We summarize our contributions as follows:
(1) We propose Diff3DS, a novel differentiable rendering framework for generating view-consistent 3D sketch from flexible inputs such as a single image or text.
(2) We are the first to represent the 3D curve as 3D rational Bézier curve and design a new differentiable rasterizer to render the perspectively projected 2D rational Bézier curves, while considering the depth order.
(3) We conduct comprehensive experiments on novel text-to-3D and image-to-3D sketch generation tasks and the results demonstrate the superiority of our method.

## 2 Related Works

#### 3D Sketch Generation

Existing research on 3D sketch generation mainly focus on interactive artistic sketch creation within 3D virtual systems [, , ], and immersive environments [, , , , , , , ]. However, these works usually require users to possess professional artistic skills and 3D operation experience, making it less user-friendly for ordinary users.
The recent 3Doodle [] is able to generate expressive 3D sketches from multiview image observations, but it is more like multiview reconstruction instead of generation and the quality of the generated results highly depends on the number of observation viewpoints.
By distilling prior knowledge from pre-trained 2D image generation models using methods like Score Distillation Sampling [, ], we achieve user-friendly and view-consistent 3D sketch generation from flexible text or single image input.

#### Differentiable Rendering of Curves

There have been several works for differentiable rendering of curves.
DiffVG [] proposes a differentiable rasterizer for the creation of vector graphics represented by 2D curves and shapes.
DRPG [] investigates differentiable rendering of 3D parametric curves and surfaces, by piecewisely approximating the continuous parametric representations with a triangle mesh.
The 3Doodle [] shares a similar differentiable rendering pipeline with us, including projecting the 3D Bézier curves to 2D Bézier curves, which are then rendered by the differentiable rasterizer from DiffVG.
However, 3Doodle performs orthogonal projection to obtain 2D Bézier curves and ignores the depth ordering between the curves.
Instead, we perform perspective projection to the 3D rational Bézier curves and designs a new DiffVG-based ratersizer which can maintain the depth ordering between the projected 2D rational Bézier curves.

#### Multimodal-Driven 3D Content Generation

Motivated by the success of the text-to-image research,
pioneering works [, ] introduce the Score Distillation Sampling (SDS) algorithm and leverage the pre-trained text-to-image models as prior knowledge to optimize 3D representations.
Subsequent works combine SDS with various differentiable 3D representations to explore the capabilities of text-to-3D object generation, such as DMTet [, ] and 3D Gaussians [, ].
Furthermore, Zero-1-to-3 [] proposes an image-guided SDS algorithm that distills the 3D-consistent priors to optimize 3D representations given the input image, and its pipeline inspires the development of subsequent image-to-3D object methods [, , , ].
Distinguishing from prior endeavors, our work stands out by focusing on generating view-consistent 3D sketch by optimizing 3D parametric curve primitives.

## 3 Preliminary

DiffVG [] proposes a differentiable rasterizer for vector graphics that supports converting 2D Bézier curves to the raster image domain and back-propagating the gradient for optimization.

Given the 2D curves parameters $\Theta$, the vector graphics scene is defined as $f(x,y,\Theta)$, and the raster image is defined as the 2D grid sampling over the space of $f(x,y,\Theta)$.
To compute the color of a 2D location $(x,y)\in\mathbb{R}^{2}$, DiffVG utilizes the inside-outside test [] to find the curves overlapped with the location initially.
Subsequently, it sorts them according to a user-specified order and calculates the color using alpha blending [].
Due to the inside-outside test, the scene function $f$ is not differentiable with respect to curve parameters.
Thus, it uses the anti-aliasing technique to make the pixel color differentiable.
By prefiltering $f$ over a convolution kernel $k$ with support $A$, sampling $I(x,y)$ at pixels can yield an alias-free image:

$$ $\displaystyle I(x,y)$ $\displaystyle=\iint_{A}k(u,v)f(x-u,y-v;\Theta)\differential{u}\differential{v}.$ (1) $$

Due to the integration over the filter support region, the focus shifts from the color value at the center point to the average pixel color. The continuous change in the average color induced by the curve movements makes the function $I(x,y)$ differentiable.

The goal is to compute the gradients of $I$ with respect to $\Theta$:

$$ $\displaystyle\frac{\partial I(x,y)}{\partial\Theta}$ $\displaystyle=\frac{\partial}{\partial\Theta}\iint_{A}k(u,v)f(x-u,y-v;\Theta) \differential{u}\differential{v}.$ (2) $$

DiffVG proposes two approaches Monte Carlo Sampling and Analytical Prefiltering to evaluate the pixel integral, which does not have a closed-form solution
in general.
We focus on the first approach which performs better.
The pixel integral can be discretized with Monte Carlo sampling []:

$$ $\displaystyle I(x,y)$ $\displaystyle=\iint_{A}k(u,v)f(x-u,y-v;\Theta)\differential{u}\differential{v} \approx\frac{1}{N}\sum\limits_{i}^{N}k(u_{i},v_{i})f(x-u_{i},y-v_{i};\Theta),$ (3) $$

but geometric discontinuities hinder the interchangeability of the integral and differential operators, which consequently obstructs the direct representation of $\frac{\partial I}{\partial\Theta}$ as a discretizable integral.
Thus, it rewrites the pixel color as the sum of integrals over multiple sub-regions $A_{i}$, and applies the Reynolds transport theorem to handle all the discontinuous changes on the boundary:

$$ $\displaystyle\frac{\partial I(x,y)}{\partial\Theta}$ $\displaystyle=\sum\limits_{i}\iint_{A_{i}(\Theta)}\frac{\partial}{\partial \Theta}g(u,v)\differential{u}\differential{v}+\sum\limits_{i}\int_{\partial A_ {i}(\Theta)}\left(\frac{\partial p(t)}{\partial\Theta}\cdot n(t)\right)g(p(t)) \differential{p(t)},$ (4) $$

where $g(u,v)$ is the multiplication of the scene function $f$ and kernel $k$ for brevity, $\partial A_{i}$ is the boundary of area $A_{i}$, $n(t)$ is the outward normal of $\partial A_{i}$, and $p(t)$ denotes the 2D points on the $\partial A_{i}$.
The first integral is responsible for the differentiation of color and the transparency, while the second term is responsible for the change of the boundaries, and the boundary integral can be estimated by the Monte Carlo estimator as:

$$ $\displaystyle\sum_{i}\int_{\partial A_{i}(\Theta)}(\nabla_{\Theta}p\cdot n)g(p (t))\differential{p(t)}\approx\frac{1}{N}\sum_{j}\frac{\left(\nabla_{\Theta}p_ {j}\cdot n_{j}\right)(g(p_{j}+\epsilon n_{j})-g(p_{j}-\epsilon n_{j}))}{P(p_{j }|c)P(c)},$ (5) $$

where $\epsilon$ is a small number and $P(p_{j}|c)P(c)$ denotes the probability density of sampling the curve $c$ and point $p_{j}$ on the boundaries.

## 4 Diff3DS

### 4.1 Overview

We propose Diff3DS, a novel differentiable rendering framework for generating view-consistent 3D sketch by optimizing 3D parametric curves.
A 3D sketch $\tilde{\Theta}$ is defined as a collection of 3D parametric curve strokes that reside in 3D space.
We particularly focus on the 3D rational Bézier curve and our derivation supports the 3D linear, quadratic, and cubic rational Bézier curves.
The theoretical trainable parameters include the control point position, curve width, and rgb-alpha color.
Follow the previous methods [, , ],
we represent each stroke by a 3D cubic Bézier curve and only optimize the control point position in our current implementation.
Moreover, weights for rational Bézier curves are disregarded to simplify the computation.
Fig. 2 illustrates the pipeline.

Diff3DS is designed with three stages.
First, to render the 3D rational Bézier curve strokes $\tilde{\Theta}$ at a given viewpoint,
we perspectively project these curves onto the camera plane (Sec. 4.2).
Then, we render the projected 2D rational Bézier curves $\Theta$ to the raster image using a customized differentiable rasterizer.
(Sec. 4.3).
Finally, we will perform back-propagation with the gradient computed from the rendered image to optimize the 3D sketch parameters (Sec. 4.4).

### 4.2 3D Rational Bézier Curve Projection

In this section, we mainly explain how to perspectively project the 3D rational Bézier curves in camera coordinates onto the 2D camera focal plane.
Given the control points $\tilde{P}_{i}=(\tilde{P}_{i}^{x},\tilde{P}_{i}^{y},\tilde{P}_{i}^{z})\in
\mathbb{R}^{3}$, the associated non-negative weights $\tilde{w}_{i}$ and the Bernstein polynomials $B_{i,n}(t)$, for $0\leq t\leq 1$, define the 3D rational Bézier curve $\tilde{p}(t)$ of degree $n$ by:

$$ $\displaystyle\tilde{p}(t)$ $\displaystyle=\sum_{i=0}^{n}\frac{B_{i,n}(t)\tilde{w}_{i}}{\sum_{j=0}^{n}B_{j, n}(t)\tilde{w}_{j}}\tilde{P}_{i}=(\tilde{x}(t),\tilde{y}(t),\tilde{z}(t)).$ (6) $$

Assume the focal plane is $z=f$, where $f$ is the focal length, the corresponding projected 2D curve $p(t)$ under the pinhole camera perspective projection can be defined as:

$$ $\displaystyle p(t)=(x(t),y(t))=(\frac{\tilde{x}(t)}{\tilde{z}(t)}f,\frac{ \tilde{y}(t)}{\tilde{z}(t)}f).$ (7) $$

Due to rational Bézier curves are projective invariant, the curve $p(t)$ still is a 2D rational Bézier curve defined by projected control points.
Given projected 2D control points
$P_{i}=(P_{i}^{x},P_{i}^{y})=(\frac{\tilde{P}_{i}^{x}}{\tilde{P}_{i}^{z}}f,
\frac{\tilde{P}_{i}^{y}}{\tilde{P}_{i}^{z}}f)\in\mathbb{R}^{2}$
and adjusted weights $w_{i}=\tilde{w}_{i}\tilde{P}_{i}^{z}$, $p(t)$ can be written as following:

$$ $\displaystyle p(t)=\sum_{i=0}^{n}\frac{B_{i,n}(t)w_{i}}{\sum_{j=0}^{n}B_{j,n}( t)w_{j}}P_{i}=(x(t),y(t)).$ (8) $$

We provide the detailed proof in Appendix B.

Figure: Figure 2: We generate the 3D sketch $\tilde{\Theta}$, represented as a set of 3D strokes, from the text or image input. We render the raster image $I$ from the random camera via our differentiable renderer (Sec. 4). Then, the pre-trained diffusion model, conditioned on the input, diffuses the rendering $I$ and predicts the pseudo ground truth $\hat{I}_{0}$. The discrepancies between $\hat{I}_{0}$ and $I$ are used to update the 3D sketch.
Refer to caption: x2.png

### 4.3 2D Rational Bézier Curves Differentiable Rasterization

This section primarily focuses on the differentiable rasterization of 2D rational Bézier curves $\Theta$ obtained through projection.

To render the projected curves and maintain their depth orders, we customize a new differentiable rasterizer based on DiffVG [].
Given a 2D pixel location $(x,y)$ and projected curves $\Theta$, we employ the inside-outside test to identify all the overlapping curves with the location.
The inside-outside test identifies a curve as overlapping with a location if the closest distance between them is less than half of the curve width, and the closest point on each overlapping curve with the location are considered as the overlapping point.
Then, we sort the overlapping point to calculate the color of the pixel location $(x,y)$ using alpha blending [].
Our sort order follows the depth order between curve primitives.
We employ the unprojected z-depth in the camera space of the overlapping point as the primary sort order.
To mitigate the z-fighting issue(^1^11https://en.wikipedia.org/wiki/Z-fighting) caused by floating-point error, we also employ a user-specified order as the secondary order.

Identifying the overlapping curve is challenging.
For the $n$th-degree rational curve $p(t)$, computing the $t$ that minimizes the distance $(p(t)-q)^{2}$ where $q=(x,y)$ equals to solve the roots of a polynomial with degree $3n-2$.
For the cubic rational curve, a 7th order polynomial needs to be solved, which doesn’t have a closed-form solution.
Inspired by DiffVG, we solve the polynomial using bisection and the Newton-Raphson method [].
The iterative solver obtains its initial guess from isolator polynomials [] – the real roots of a 7th order polynomial can be isolated by the roots of two cubic auxiliary polynomials.
Please refer to Appendix C for more details.

After obtaining the 2D curve scene function $f(x,y,\Theta)$,
the rendered raster image $I$ will be calculated using the Monte Carlo Sampling strategy (Sec. 3)
with Eq. 3.
Fig. 3 shows an example.

Figure: Figure 3: A rasterized result contains 3 quadratic rational Bézier curves and a line. All curves share the same control point positions in pixel space and the depths but with different weights. Our rasterizer accurately renders the curves and faithfully maintains occlusions according to the depth order. The user-specified sort order results in the pixel at position (1023, 0) being the color of Curve C, as it has a higher priority than the other overlapping curve points with the same depth.
Refer to caption: x3.png

### 4.4 Gradients Back-propagation

The rendering framework is fully differentiable since the projection and rasterizer are differentiable.
Thus, the gradients of $I$ with respect to 3D curves parameter $\tilde{\Theta}$ can be calculated with the chain rule:
$\frac{\partial I(x,y)}{\partial\tilde{\Theta}}=\frac{\partial I(x,y)}{\partial
\Theta}\frac{\partial\Theta}{\partial\tilde{\Theta}},$
where $\frac{\partial I}{\partial\Theta}$ denotes the gradients of $I$ with respect to 2D projected curves $\Theta$ that can be computed using Eq. 4 and Eq. 5, and
$\frac{\partial\Theta}{\partial\tilde{\Theta}}$ denotes the gradients of projected 2D
curves $\Theta$ with respect to 3D curves $\tilde{\Theta}$.
$\frac{\partial I(x,y)}{\partial\Theta}$ calculates the gradients of control point positions, color, and width for projected 2D curves.
The position gradients of 3D curve control points are calculated with $\frac{\partial\Theta}{\partial\tilde{\Theta}}$, which is based on the gradients of the projected 2D control points.
The color and width gradients of 3D curves directly inherit the corresponding gradients of the projected 2D curves.

## 5 Multimodal-Driven 3D Sketch Generation

Diff3DS supports end-to-end optimization of 3D sketch under flexible supervisions.
By employing the Score Distillation Sampling algorithm, we propose the promising tasks of text-to-3D sketch and image-to-3D sketch, with the goal of generating 3D sketch from flexible text or single image input.

### 5.1 Text-to-3D Sketch

Pioneering works [, ] propose the Score Distillation Sampling (SDS) algorithm, utilizing the prior of a pre-trained 2D text-to-image model to optimize 3D representations.
By integrating SDS with Diff3DS, we distill the diffusion prior to generate the 3D sketch $\tilde{\Theta}$ from the input text.
Given a pre-trained diffusion model $\epsilon_{\phi}$, text embedding $y$, rendered image $I$ with our rasterizer $\mathcal{R}$ and the noise timestep $t$,
image $I$ will be added with noise to obtain a noisy image $I_{t}$:
$I_{t}=\sqrt{\bar{\alpha}_{t}}I+\sqrt{1-\bar{\alpha}_{t}}\epsilon,$
where $\epsilon\sim\mathcal{N}(0,I)$ and $\bar{\alpha}_{t}$ is the cumulative product of scaling at timestep $t$.
Then the gradient of SDS loss on $\tilde{\Theta}$ is given by:

$$ $\nabla_{\tilde{\Theta}}\mathcal{L}_{SDS}(\phi,I=\mathcal{R}(\tilde{\Theta}))= \mathbb{E}_{t,\epsilon}\left[\omega(t)(\hat{\epsilon}_{\phi}(I_{t};y,t)- \epsilon)\frac{\partial I}{\partial\tilde{\Theta}}\right],$ (9) $$

where $\omega(t)$ is a weight based on the timestep $t$, and the predicted noise sampled by classifier-free guidance (CFG) [] with weight $\lambda$ denotes as:

$$ $\hat{\epsilon}_{\phi}(I_{t};y,t)=\epsilon_{\phi}(I_{t};\emptyset,t)+\lambda( \epsilon_{\phi}(I_{t};y,t)-\epsilon_{\phi}(I_{t};\emptyset,t)).$ (10) $$

By deriving the pseudo ground truth image $\hat{I}_{0}$ with one-step denoising:

$$ $\displaystyle\hat{I}_{0}=\frac{1}{\sqrt{\bar{\alpha}_{t}}}(I_{t}-\sqrt{1-\bar{ \alpha}_{t}}\hat{\epsilon}_{\phi}(I_{t};y,t)),$ (11) $$

we can transform Eq. 9 into an equivalent form as follows:

$$ $\displaystyle\hskip 1.42262pt\nabla_{\tilde{\Theta}}\mathcal{L}_{SDS}(\phi,I= \mathcal{R}(\tilde{\Theta}))$ $\displaystyle=\mathbb{E}_{t,\epsilon}\left[\omega(t)\frac{\sqrt{\bar{\alpha}_{ t}}}{1-\sqrt{\bar{\alpha}_{t}}}(I-\hat{I}_{0})\frac{\partial I}{\partial\tilde {\Theta}}\right],$ (12) $$

which can be regarded as matching the input view $I$ with predicted $\hat{I}_{0}$.

Similar with [], due to the bias of 2D text-to-image model, we encounter serious view inconsistency problem known as the Janus problem.
This problem persists even when directional prompts (e.g., "the front view of…") are used to distinguish between different views.
To mitigate this issue, we integrate MVDream [] with our framework, which is able to generate 3D-aware four view images using input text and camera poses.

Figure: Figure 4: Pseudo ground truth $\hat{I}_{0}$ visualization. As the CFG weight $\lambda$ decreases, the effective supervision region of $\hat{I}_{0}$ also decreases.
Refer to caption: x4.png

The impact of the CFG weight $\lambda$ on the optimization process has received attention in existing text-to-3D methods.
By visualizing the pseudo ground truth $\hat{I}_{0}$ after denoising with different levels of weight $\lambda$ at large noise timesteps in Fig. 4, we find that as $\lambda$ decreases, the effective supervision region of $\hat{I}_{0}$ decreases, hereby impacting the final shape of results.

### 5.2 Image-to-3D Sketch

Zero-1-to-3 [] proposes an image-guided SDS method that conditions on the input view image $\tilde{I}$ and relative camera extrinsic ($R$, $T$), based on its view-conditioned model.
Our framework is able to generate 3D sketch $\tilde{\Theta}$ from the input reference image $\tilde{I}$ by distilling the 3D consistent prior:

$$ $\nabla_{\tilde{\Theta}}\mathcal{L}_{SDS}(\phi,I=\mathcal{R}(\tilde{\Theta}))= \mathbb{E}_{t,\epsilon}\left[\omega(t)(\hat{\epsilon}_{\phi}(I_{t};\tilde{I},R ,T,t)-\epsilon)\frac{\partial I}{\partial\tilde{\Theta}}\right].$ (13) $$

### 5.3 Dynamic Noise Deletion

We find that our framework is affected by the issue of gradient sparsity in the Monte Carlo Sampling strategy.
In Reynolds’ formula, the gradient from pixel space is determined by the curve boundaries.
This results in the loss of information from pixels that are not incident to those boundaries.
In cases where significant displacements of curves are required, the optimization tends to get trapped in a local optimum state: curves contract the boundaries to minimize the loss function.
During our training process, some curves tend to contract to extremely small lengths as noises.
To obtain clean results, we propose the Dynamic Noise Deletion strategy.
We discretize each curve into $n$ line segments ($n=20$) and accumulate their lengths as the total length of the curve.
The curve with lengths below a specified threshold will be considered as noise and removed dynamically.

### 5.4 Time Annealing Schedule

To further enhance the performance of the results, we employ a time annealing schedule similar with [, , , ].
During the optimization process, we gradually adjust the maximum and minimum time step for SDS in a linear manner.
The large noise timestep focuses on aligning the curves with the semantic content of the input text or image during the initial training phase.
Conversely, the small noise timestep employed during the later training phase focuses on further enhancing the details.

Figure: Figure 5: Qualitative results of the text-to-3D sketch task. Existing text-to-3D methods fail to generate sketch-style results, even with the addition of the "sketch in black and white, line drawing" suffix.
Refer to caption: x5.png

## 6 Experiments

### 6.1 Implementation Details

We implement our rendering framework in C++/CUDA with a PyTorch interface [], and follow
the STEP standard (ISO 10303-21) [] to format and save our 3D curve results. In the experiment, a user-specified number of curves will be randomly initialized within a sphere of radius 1.5.
We randomly sample the camera position using the radius from 1.8 to 2.0, with the azimuth in the range of -180 to 180 degrees, the elevation in the range of 0 to 30 degrees and the field of view (fov) of 60 degrees.
Notice that Stable-Zero123 abnegates the relative distance for simplifying, we fix its radius at $2.0$.
For the pre-trained model, we apply Stable Diffusion 2.1 [] and MVDream [] for the text-to-3D sketch task, with the CFG weight of $50$.
And we apply Zero-1-to-3 [] and Stable-Zero123 [] for the image-to-3D sketch task, with the CFG weight of $7.5$ followed [, ] .
For all tasks, the total number of training steps is 4000.
Starting from step 2000, we dynamically delete the noise every 100 steps.
The training process requires 1 hour for the text-to-3D sketch task and 2 hours for the image-to-3D sketch task on a single NVIDIA A10 GPU with a batch size of 4.

### 6.2 Text-to-3D Sketch

#### Baselines

To the best of our knowledge, we are the first text-to-3D sketch method.
We select the existing text-to-3D object methods DreamGaussian [] and MVDream [] as compared baselines, and further choose the text-to-2D sketch method DiffSketcher [] as the additional reference.

#### Quantitative Comparisons

We collect 35 text prompts from previous works [, ] and websites.
To encourage baseline methods to generate sketch-style results, we append the suffix "sketch in black and white, line drawing" to each prompt inputted to the baselines.
We measure the CLIP [] text-image similarity (CLIP-Score^T) metric to evaluate the semantic coherence between the input text prompt and the rendered images.
For each generated 3D content, we compute the average metric across four orthogonal view images relative to the input text prompt.
Table 2 shows the quantitative results of the text-to-3D sketch task.
DiffSketcher achieves the best performance, while our method outperforms all the text-to-3D baselines.

#### Qualitative Comparisons

Fig. 5 shows the qualitative results of the text-to-3D sketch task.
We compared the 3D results of the text-to-3D baseline method with our method, and further provide the 2D results of DiffSketcher.
Compared text-to-3D baselines utilize Stable Diffusion [] as the prior provider but do not fully inherit its capability. They struggle to generate the sketch-style content, even with the additional suffix.
Our method and DiffSketcher can generate the 3D sketch and 2D sketch similar with the input text.

**Table 1: Quantitative results of the text-to-3D sketch task. The CLIP-Score^T metric is shown.**
| Method | CLIP-Score^T $\uparrow$ |  |
| --- | --- | --- |
| ViT B/16 | ViT B/32 |  |
| DiffSketcher [] | 33.64 | 33.98 |
| DreamGaussian [] | 26.25 | 25.12 |
| MVDream [] | 28.39 | 27.08 |
| Our | 30.35 | 30.00 |

### 6.3 Image-to-3D Sketch

#### Baselines

To the best of our knowledge, we are the first to generate 3D sketch from a single image.
The closest works to ours are Neural Edge Fields (NEF) [] and 3Doodle [], which generate 3D curves from multiview images.
However, since 3Doodle has not released the code, we select NEF for comparison.
NEF reconstructs 3D curves from the 2D multiview edge maps estimated from PidiNet [].
We also choose the image-to-2D sketch method CLIPasso [] as the additional reference.

#### Quantitative Comparisons

We collect 25 images generated with Imagine [], and measure the CLIP-Score [] visual similarity (CLIP-Score^I) metric to evaluate the semantic coherence between the input reference image and the rendered images.
For each generated 3D content, we render four orthogonal view images to calculate the average metrics with the reference image.
Table 2 illustrates that CLIPasso performs the best, while our method outperforms NEF.

Figure: Figure 6: Qualitative results of the image-to-3D sketch task. NEF fails to exact reasonable 3D curves from the input image.
Refer to caption: x6.png

#### Qualitative Comparisons

Fig. 6 shows the qualitative results of the image-to-3D sketch task.
As a reconstruction method, NEF fails to extract reasonable 3D curves from a single input image because it lacks the ability to predict the unobserved regions.
Our method and CLIPasso can generate the 3D sketch and 2D sketch, respectively, conditioned on the input image.

### 6.4 Ablations and Analysis

Figure: Figure 7: Illustration of the optimization process.
Refer to caption: x7.png

#### Optimization Process Visualization

Fig. 7 illustrates the visualization of the training optimization process.
The optimization starts with randomly initialized curves.
As time progresses, curves gradually optimize to resemble the input text or image.
Dynamic Noise Deletion intervenes during the middle stage of training to remove the noise generated during the training process.

#### Effect of Hyperparameters

Fig. 8 illustrates the effect of hyperparameters.
Fig. 8 (a)
illustrates the effect of different CFG weights $\lambda$ in the text-to-3D sketch task.
A larger weight leads to a stronger shape prior, while a smaller weight results in shape degradation.
Fig. 8 (b)
illustrates the effect of various pre-trained models. The Janus Problem easily arises when using Stable Diffusion, and MVDream partially mitigates this issue (e.g., the bicycle with three wheels in the bird’s-eye view). Then Zero-1-to-3 is not sufficiently 3D consistent in certain examples (e.g., the upper part of the scissors is bent in the side view).
Fig. 8 (c)
illustrates that increasing the initial curve number can enhance the details of results, even though the final curve number is automatically determined by Dynamic Noise Deletion. Too few curves may fail to represent a complete 3D object.

Figure: (a) CFG weight.
Refer to caption: x8.png

#### Ablation Study of Designed Components

We evaluate the contributions of the Time Annealing Schedule and Dynamic Noise Deletion components, as shown in Fig. 9.
The omission of Time Annealing Schedule affects the conformity of the generated results with the semantic meaning of the input text or the level of detail in the reference image, and the omission of Dynamic Noise Deletion leads to the retention of some noise in the results.

Figure: Figure 9: Ablation study on Time Annealing Schedule (TAS) and Dynamic Noise Deletion (DND).
Refer to caption: x11.png

## 7 Conclusions

In this paper, we propose Diff3DS, a novel differentiable rendering framework for generating view-consistent 3D sketch by optimizing 3D parametric curves under various supervisions.
By employing the recent Score Distillation Sampling (SDS) to distill prior knowledge from pre-trained 2D image
generation model, we achieve 3D sketch generation from the flexible text or single image input.
Extensive experiments have demonstrated the potential of our framework.

#### Limitations and Future Work

First, Diff3DS inherits the sparse gradient issue from DiffVG and can only optimize continuous parameters.
Moreover, our downstream task assumption does not distinguish between the view-independent curve (e.g., feature lines) and the view-dependent curve (e.g., contours of smooth surface boundaries), limiting the expressive capacity of the overall shape of a 3D object, which is extensively discussed in 3Doodle.
Incorporating the diverse curve representations of 3Doodle is a promising solution.
Also, one future direction is to extend our method for scene-level generation task, such as text-to-3D scene sketch.

## Supplementary

## Overview

This supplementary material is structured into multiple sections that offer further details and analysis pertaining to our Diff3DS research.
Concretely, it will hone in on the topics that follow:

- $\bullet$
In section A,we provide the implementation details of Diff3DS.
- $\bullet$
In section B, we provide a detailed proof for the 3D rational Bézier curve projection.
- $\bullet$
In section C, We provide a detailed analysis for finding the closest distance between a point and a rational Bézier curve.
- $\bullet$
In section D, we provide more high quality sketch results.

## Appendix A Implementation Details

In the experiment, a user-specified number of curves will be randomly initialized within a sphere of radius 1.5.
The default curve number is set to 56.
We randomly sample the camera position using the radius from 1.8 to 2.0, with the azimuth in the range of -180 to 180 degrees, the elevation in the range of 0 to 30 degrees and the field of view (fov) of 60 degrees.
Notice that Stable-Zero123 abnegates the relative distance for simplifying (^2^22https://github.com/threestudio-project/threestudio/issues/360), we fix its radius at $2.0$.

For the pre-trained model, we apply Stable Diffusion 2.1 [] and MVDream [] for the text-to-3D sketch task, with the CFG weight of $50$.
And we apply Zero-1-to-3 [] and Stable-Zero123 [] for the image-to-3D sketch task, with the CFG weight of $7.5$ followed by [, ] .

For all tasks, the total number of training steps is 4000.
Starting from step 2000, we dynamically delete the noise every 100 steps.
The training process requires 1 hour for the text-to-3D sketch task and 2 hours for the image-to-3D sketch task on a single NVIDIA A10 GPU with a batch size of 4.
To optimize the control point positions, we use the Adam optimizer and set the learning rate of the optimizer to 0.002.
For the time annealing schedule, we prefer to decrease the maximum and minimum time steps from 0.85 to 0.3 and 0.1, respectively, over the first 3600 steps.
Due to limited resources, not all possible combinations of the hyper-parameters related to the time steps have been fully explored, and there may be other configurations that could produce better results.

## Appendix B 3D Rational Bézier Curve Projection

Given the control points $\tilde{P}_{i}=(\tilde{P}_{i}^{x},\tilde{P}_{i}^{y},\tilde{P}_{i}^{z})\in
\mathbb{R}^{3}$, the associated non-negative weights $\tilde{w}_{i}$ and the Bernstein polynomials $B_{i,n}(t)$, for $0\leq t\leq 1$, define a $n$th-degree 3D rational Bézier curve $\tilde{p}(t)$ by:

$$ $\displaystyle\tilde{p}(t)$ $\displaystyle=\sum_{i=0}^{n}\frac{B_{i,n}(t)\tilde{w}_{i}}{\sum_{j=0}^{n}B_{j, n}(t)\tilde{w}_{j}}\tilde{P}_{i}=(\tilde{x}(t),\tilde{y}(t),\tilde{z}(t)).$ $$

Assume the focal plane is $z=f$, where $f$ is the focal length, the corresponding projected 2D curve $p(t)$ under the pinhole camera perspective projection can be defined as:

$$ $\displaystyle p(t)=(x(t),y(t))=(\frac{\tilde{x}(t)}{\tilde{z}(t)}f,\frac{ \tilde{y}(t)}{\tilde{z}(t)}f).$ $$

Due to rational Bézier curves are projective invariant, the 2D curve $p(t)$ still is a rational Bézier curve defined by projected control points.
Given projected 2D control points

$$ $\displaystyle P_{i}=(P_{i}^{x},P_{i}^{y})=(\frac{\tilde{P}_{i}^{x}}{\tilde{P}_ {i}^{z}}f,\frac{\tilde{P}_{i}^{y}}{\tilde{P}_{i}^{z}}f)\in\mathbb{R}^{2},$ $$

$p(t)$ can be written as a 2D rational Bézier curve with adjusted weights $w_{i}=\tilde{w}_{i}\tilde{P}_{i}^{z}$:

$$ $\displaystyle p(t)$ $\displaystyle=(\frac{\tilde{x}(t)}{\tilde{z}(t)}f,\frac{\tilde{y}(t)}{\tilde{z }(t)}f)$ $\displaystyle=(\frac{\sum_{i=0}^{n}B_{i,n}(t)\tilde{w}_{i}\tilde{P}_{i}^{x}}{ \sum_{j=0}^{n}B_{j,n}(t)\tilde{w}_{j}\tilde{P}_{j}^{z}}f,\frac{\sum_{i=0}^{n}B _{i,n}(t)\tilde{w}_{i}\tilde{P}_{i}^{y}}{\sum_{j=0}^{n}B_{j,n}(t)\tilde{w}_{j} \tilde{P}_{j}^{z}}f)$ $\displaystyle=(\sum_{i=0}^{n}\frac{B_{i,n}(t)(\tilde{w}_{i}\tilde{P}_{i}^{z})} {\sum_{j=0}^{n}B_{j,n}(t)(\tilde{w}_{j}\tilde{P}_{j}^{z})}\left(\frac{\tilde{P }_{i}^{x}}{\tilde{P}_{i}^{z}}f\right),\sum_{i=0}^{n}\frac{B_{i,n}(t)(\tilde{w} _{i}\tilde{P}_{i}^{z})}{\sum_{j=0}^{n}B_{j,n}(t)(\tilde{w}_{j}\tilde{P}_{j}^{z })}\left(\frac{\tilde{P}_{i}^{y}}{\tilde{P}_{i}^{z}}f\right))$ $\displaystyle=(\sum_{i=0}^{n}\frac{B_{i,n}(t)w_{i}}{\sum_{j=0}^{n}B_{j,n}(t)w_ {j}}P_{i}^{x},\sum_{i=0}^{n}\frac{B_{i,n}(t)w_{i}}{\sum_{j=0}^{n}B_{j,n}(t)w_{ j}}P_{i}^{y})$ $\displaystyle=(x(t),y(t))$ $\displaystyle=\sum_{i=0}^{n}\frac{B_{i,n}(t)w_{i}}{\sum_{j=0}^{n}B_{j,n}(t)w_{ j}}P_{i}.$ $$

## Appendix C Closest Distance between a Point and a Rational Bézier Curve

Given a point $q\in\mathbb{R}^{2}$ and a $n$th-degree rational Bézier curve

$$ $p(t)=\frac{\sum_{i=0}^{n}B_{i,n}(t)w_{i}P_{i}}{\sum_{j=0}^{n}B_{j,n}(t)w_{j}},$ $$

we want to find the closest distance between them and the closest point $t^{\star}$ on the curve $p$.
The closest distance is defined as $(p(t^{\star})-q)^{2}$, and $t^{\star}$ is defined as:

$$ $t^{\star}=\arg_{t}\min(p(t)-q)^{2}.$ $$

To solve the $t^{\star}$, we take the derivative of the squared distance
with respect to $t$ and set it to zero:

$$ $2(p(t)-q)p^{\prime}(t)=\rho(t)=0,$ $$

which equal to solve the roots of a polynomial with degree $3n-2$.
For the quadratic rational curve and the cubic rational curve, a 4th order polynomial and a 7th order polynomial need to be solved respectively. We provide the detail separately in section C.1 and section C.2.

To solve all real roots of these polynomials, inspired by DiffVG, we solve the polynomial using bisection and the Newton-Raphson method [].
The Newton-Raphson solver obtains its initial guess for intervals from isolator polynomials [].
For the polynomial $\rho(t)$ with two adjacent real roots $t_{1}$ and $t_{2}$,
given any two lower order other polynomials $b(t)$ and $c(t)$, define

$$ $a(t)=b(t)\rho^{\prime}(t)+c(t)\rho(t),$ $$

where $\rho^{\prime}(t)$ is the derivative,
[] proofs that $a(t)$ or $b(t)$ must has at least one real root in the closed interval $[t_{0},t_{1}]$.
Since $\rho(t_{0})=\rho(t_{1})=0$, we know $a(t_{0})a(t_{1})=b(t_{0})b(t_{1})\rho^{\prime}(t_{0})\rho^{\prime}(t_{1})$.
Since $t_{0}$ and $t_{1}$ are roots of $\rho$ and $\rho^{\prime}(t_{0})\rho^{\prime}(t_{1})\leq 0$.
Thus either $a(t_{0})a(t_{1})\leq 0$ or $b(t_{0})b(t_{1})\leq 0$.
Please see the original paper for discussions on multiple roots.

After solving all the real roots of $a$ and $b$ within the $(0,1)$, the intervals for finding the real roots of $\rho$ are then determined. We can then use bisection and the Newton-Raphson method to find all the roots of $\rho$ within these intervals.
The key operation is the selection of the polynomials $b(t)$ and $c(t)$, [] prefer to find the $b(t)$ and $c(t)$ which follow $\mathrm{Degree}(a)+\mathrm{Degree}(b)=\mathrm{Degree}(\rho)-1$.
In the upcoming paragraphs, we will discuss the cases of the 4th order polynomial and the 7th order polynomial in
section C.3 and section C.4.

### C.1 Closest Distance of Quadratic Rational Curve Case

The quadratic rational curve $p(t)$ can be reorganized using variables $A$ through $F$ as follows:

$$ $\displaystyle p(t)$ $\displaystyle=\frac{(1-t)^{2}w_{0}P_{0}+2(1-t)tw_{1}P_{1}+t^{2}w_{2}P_{2}}{(1- t)^{2}w_{0}+2(1-t)tw_{1}+t^{2}w_{2}}=\frac{At^{2}+Bt+C}{Dt^{2}+Et+F},$ $$

we can further calculate $p(t)-q$ and $p^{\prime}(t)$ as:

$$ $\displaystyle p(t)-q=\frac{(A-Dq)t^{2}+(B-Eq)t+(C-Fq)}{Dt^{2}+Et+F}$ $\displaystyle p^{\prime}(t)=\frac{(AE-BD)t^{2}+(2AF-2CD)t+(BF-CE)}{(Dt^{2}+Et+ F)^{2}}$ $$

By reorganizing them using variables $a$ through $f$:

$$ $\displaystyle p(t)-q=\frac{at^{2}+bt+c}{Dt^{2}+Et+F}$ $\displaystyle p^{\prime}(t)=\frac{dt^{2}+et+f}{(Dt^{2}+Et+F)^{2}},$ $$

we can rewrite the target polynomial $\rho(t)$ as below:

$$ $\displaystyle\rho(t)$ $\displaystyle=2(p(t)-q)p^{\prime}(t)=2\frac{(at^{2}+bt+c)(dt^{2}+et+f)}{(Dt^{2 }+Et+F)^{3}},$ $$

whose numerator is a 4th order polynomial in the variable $t$.
Note that the denominator of $\rho(t)$ is strictly positive, due to the definition of the rational Bézier curve.
Therefore, finding the real roots of the equation $\rho(t)=0$ is equivalent to finding the real roots of the 4th order polynomial in the numerator.

### C.2 Closest Distance of Cubic Rational Curve Case

The cubic rational curve $p(t)$ can be reorganized using variables $A$ through $H$ as follows:

$$ $\displaystyle p(t)$ $\displaystyle=\frac{(1-t)^{3}w_{0}P_{0}+3(1-t)^{2}tw_{1}P_{1}+3(1-t)t^{2}w_{2} P_{2}+t^{3}w_{3}P_{3}}{(1-t)^{3}w_{0}+3(1-t)^{2}tw_{1}+3(1-t)t^{2}w_{2}+t^{3}w _{3}}$ $\displaystyle=\frac{At^{3}+Bt^{2}+Ct+D}{Et^{3}+Ft^{2}+Gt+H},$ $$

we can further calculate $p(t)-q$ and $p^{\prime}(t)$ as:

$$ $\displaystyle p(t)-q=\frac{(A-Eq)t^{3}+(B-Fq)t^{2}+(C-Gq)t+(D-Hq)}{Et^{3}+Ft^{ 2}+Gt+H}$ $\displaystyle p^{\prime}(t)=\frac{3A^{2}+2Bt+C}{Et^{3}+Ft^{2}+Gt+H}-\frac{(3Et ^{2}+2Ft+G)(At^{3}+Bt^{2}+Ct+D)}{(Et^{3}+Ft^{2}+Gt+H)^{2}}$ $$

By reorganizing them using variables $a$ through $i$:

$$ $\displaystyle p(t)-q=\frac{at^{3}+bt^{2}+ct+d}{Et^{3}+Ft^{2}+Gt+H}$ $\displaystyle p^{\prime}(t)=\frac{et^{4}+ft^{3}+gt^{2}+ht+i}{(Et^{3}+Ft^{2}+Gt +H)^{2}},$ $$

we can write the target polynomial $\rho(t)$ as below:

$$ $\displaystyle\rho(t)$ $\displaystyle=2(p(t)-q)p^{\prime}(t)=2\frac{(at^{3}+bt^{2}+ct+d)(et^{4}+ft^{3} +gt^{2}+ht+i)}{(Et^{3}+Ft^{2}+Gt+H)^{3}},$ $$

whose numerator is a 7th order polynomial in the variable $t$.
Note that the denominator of $\rho(t)$ is strictly positive, due to the definition of the rational Bézier curve.
Therefore, finding the real roots of the equation $\rho(t)=0$ is equivalent to finding the real roots of the 7th order polynomial in the numerator.

### C.3 Isolator Polynomials of the 4th order Polynomial Case

We aim to isolate the real roots of the 4th order polynomial $\rho(t)$
by the roots of two lower order polynomials $a(t)$ and $b(t)$.
To keep $\mathrm{Degree}(a)+\mathrm{Degree}(b)=\mathrm{Degree}(\rho)-1$,
we select $a(t)$ as the 2nd order polynomial and $b(t)$ as the linear polynomial.

Define $\rho(t)$ and $\rho^{\prime}(t)$ as follows:

$$ $\displaystyle\rho(t)$ $\displaystyle=t^{4}+Bt^{3}+Ct^{2}+Dt+E$ $\displaystyle\rho^{\prime}(t)$ $\displaystyle=4t^{3}+3Bt^{2}+2Ct+D,$ $$

and we know $a(t)=c(t)\rho(t)-b(t)\rho^{\prime}(t).$
Let $c(t)=1$, $a(t)$ can be written as $a(t)=\rho(t)-b(t)\rho^{\prime}(t)$
By doing a long division between $\rho$ and $\rho^{\prime}$, we obtain the isolator polynomials as follows:

$$ $\displaystyle a(t)$ $\displaystyle=(\frac{C}{2}-\frac{3B^{2}}{16})t^{2}+(\frac{3D}{4}-\frac{CB}{8}) t+(e-\frac{DB}{16})$ $\displaystyle b(t)$ $\displaystyle=\frac{1}{4}t+\frac{B}{16}$ $$

### C.4 Isolator Polynomials of the 7th order Polynomial Case

We aim to isolate the real roots of the 7th order polynomial $\rho(t)$
by the roots of two lower order polynomials $a(t)$ and $b(t)$.
To keep $\mathrm{Degree}(a)+\mathrm{Degree}(b)=\mathrm{Degree}(\rho)-1$,
we select $a(t)$ and $b(t)$ both be the cubic polynomial.

Define $\rho(t)$ and $\rho^{\prime}(t)$ as follows:

$$ $\displaystyle\rho(t)$ $\displaystyle=t^{7}+p_{6}t^{6}+....+p_{1}t+p_{0}$ $\displaystyle\rho^{\prime}(t)$ $\displaystyle=7t^{6}+6p_{6}t^{5}+....+p_{1},$ $$

and we know $a(t)=c(t)\rho(t)-b(t)\rho^{\prime}(t).$
We wish $a$ and $b$ to both be cubic polynomials. Note that $b(t)p^{\prime}(t)$ is a 9th order polynomial, so we assume $c(t)$ to be a 2nd order polynomial to ensure that $c(t)p(t)$ is a 9th order polynomial too.
Let $b(t)=(t^{3}+Ct^{2}+Dt+E)$ and $c(t)=(7t^{2}+At+B)$, we can obtain two 9th order polynomials, and their difference a(x) is a cubic polynomial:

$$ $\displaystyle c(t)\rho(t)$ $\displaystyle=(t^{7}+p_{6}t^{6}+....+p_{1}t+p_{0})(7t^{2}+At+B)$ $\displaystyle b(t)\rho^{\prime}(t)$ $\displaystyle=(7t^{6}+6p_{6}t^{5}+....+p_{1})(t^{3}+Ct^{2}+Dt+E).$ $$

By setting the coefficients of the 8th to 4th order terms of $a(t)$ to 0, we can obtain a system of 5 linear equations in variables A through E.
By solving this system of equations, we can determine the values of variables A through E. We can then proceed to find $b(t)$, $c(t)$, and $a(t)$.
Please refer to our code for the specific details.

## Appendix D More High Quality Results

### D.1 More Results of the Text-to-3D Sketch Task

Figure: Figure 10: More results of the text-to-3D sketch task.
Refer to caption: x12.png

### D.2 More Results of the Image-to-3D Sketch Task

Figure: Figure 11: More results of the image-to-3D sketch task
Refer to caption: x13.png

### D.3 More 3D Sketch Results Rendered using Blender [ 9 ]

Figure: Figure 12: More rendered 3D sketch results.
Refer to caption: x14.png

Figure: Figure 13: More rendered 3D sketch results.
Refer to caption: x15.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 5
presentation: 4
questions: It appears from Figure 6 that, while your approach produces more 3D consistent results than 3Doodle, it less closely captures the structure of the guiding image. Do you have a sense of why that might be? Is it due to the fact that Diff3DS is more 3D consistent and thus is less able to overfit to exact structural details? Can this be improved?
soundness: 3
strengths: Strengths:
- Through the use of score distillation sampling for supervision, this work is the first to enable text/image driven 3D stroke generation from scratch (as opposed to existing works which simplify / abstract / sketchify an existing 3D representation defined by multi-veiw images).
- The novel differentiable rasterization technique introduced by this work allows for more accurate depths when rasterizing 3D curves.
- The paper is well written and organized.
summary: The authors present Diff3DS, a framework for optimizing 3D strokes using text or image guidance. This work introduces a novel extension of DiffVG that allows users to differentiably rasterize 2D projections of 3D parametric curves with more accurate depths / occlusions based on the 3D location of the curves. Using this new rasterization technique, Diff3DS applies SDS supervision to generate view-consistent 3D sketches using text or image guidance. This work validates its approach through ablations of key components and comparisons to other similar works. Diff3DS shows improved performance over other methods in both automated metrics and a perceptual user study. The paper is well written and easy to follow. The method is explained in sufficient detail and design decisions are properly motivated. Informative figures clearly showcase the contributions of this work (i.e. Figure 2 nicely demonstrates this ability to have non-uniform ordering between curves). The provided experiments validate claims made in the paper.
weaknesses: Weaknesses:
- The main contribution of this work appears to be the improved differentiable rasterization approach that enables more accurate depths during rasterization. However, it is not clear that this component is responsible for improved performance. Figure 11 ablates the impact of this rasterization technique on 3Doodle. However, the differences are quite minor and at this level of abstraction, the depths of these curves seem less important to the visual coherence of the object.
- The comparisons currently shown do not use strong enough / similar enough baselines. Some of the comparisons shown are to standard (non-sketch) generative approaches. Showing these is fine, especially given the lack of existing work on text driven 3D consistent sketch generation. However, it feels that using score distillation guidance + DiffVG should still produce strong results for text/image guided curve generation. This is an important baseline and should be compared to in the paper.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: * Why care about z-ordering of the curves? In which situation is it relevant?
* Figure 4: The meaning is unclear. Is this supposed to represent a pseudo-ground truth image? How does it relate to gradient supervision?
* In which situation are 3D sketches useful?
soundness: 3
strengths: * Covers multiple aspects of differentiable rendering.
* Extends DiffVG, a differentiable rasterizer for 2D vector graphics, to 3D Bézier curves.
* Presents a relatively simple and elegant pipeline, clearly summarized in Figure 3.
* Includes small but useful techniques that are well explained and analyzed:
    * Time annealing for the Score Distillation Sampling (SDS).
    * Curve filtering to reduce noise.
summary: The paper proposes a framework for generating 3D sketches from either a text prompt or a single image. It is based on differentiable rendering of 3D Bézier curves and applies supervision from pretrained image diffusion models from random viewpoints.
weaknesses: * The z-ordering of Bézier curves in Section 4.3 is the most technically advanced contribution. But the motivation is unclear: all sketches consist of black strokes, and intersections are rare at the image scale. Why care about correctly ordering the curves? Figure 11’s ablation shows only very minor visual differences. The effect of this ordering on 3D sketch generation is also left unclear.
* In the related works section, the motivations against existing 3Doodle approaches seem weak:
    * Issues with perspective projections.
    * Use of z-ordering for handling colored intersections.
* The usefulness and relevance of 3D sketches as shape representation is not fully clarified.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: No more questions.
soundness: 3
strengths: - In general, the paper is very well-presented. 
- To make 3D sketch rendering differienable is important and challenge. The paper proposed a novel one which enables the maintaining of the depth ordering. I also believe such rendering framework could benefit some other tasks. 
- The results seem very good. It's far better than existing ones visually.
summary: Nowadays, there are a lot of works conducting optimization of 3D models from 2D diffusion priors with SDS loss. Insteading of optimizing 3D models, this work aims to output 3D sketch which is also an important 3D format in related areas. To do so, a novel differatiable rendering framework is proposed. Experimental results validate the effectiveness of the proposed methods. Compared with existing differenatiable curve rendering methods, such as 3Doodle, the proposed method can maintain depth ordering which is very helpful for the opimization.
weaknesses: There are also some issues which can be improved. 
1) Disscussions on the practical usage of 3D sketches can be improved. After the first glance of this paper, my primary curiousity is from the usage of 3D sketches. In the paper, the authors just mentioned "widely used" but did not provide examples. For me, it is really hard to imagine the usage of such 3D format. When thinking this in deep, I think 3D sketch/line can be an abstract representation of 3D structure, which can be an intermediate representation for some tasks like 3D reconstruction. In this senario, the differentiable rendering will also be very important. More discussions on this are strongly encouraged to make the application senarios more clear. More specifically, I think a subsection is needed to discuss those applications of 3D sketches. 

2) The evaluation is somewhat weak. As mentioned in the title and intro, view-consistency is the target of the proposed method. Howerver, there are only one user study to measure the view-consistency which is weak. The authors is encouraged to present some numeric results of view-consistency. For example, given two adjacent views, we can do a warping based on the optical flow estimation and then calculate the alignment error. If two views are consistent, the error should be minor. This metric can be compared with some baselines such as zero-123. 

3) I also have a concern about the number of curves. It seems the number of curves is a parameter needs to be set manually. This will increase the burden of the users. If it is so, please include this discussion into the limitation part.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: 1. In some examples, such as the phoenix, the 3D curves appear insufficient to fully capture the object's shape, and the curves still seem somewhat disorganized. Can this be solved with a higher number of 3D curves?

2. The method initializes curves within a fixed-size sphere. How heavily does this initialization impact the final optimization?

3. Why not consider optimizing additional parameters, such as the width, color, or transparency of the 3D curves? Could optimizing these properties lead to significantly better results?
soundness: 3
strengths: 1. The paper introduces a novel framework for 3D sketch generation, which, to the best of my knowledge, is the first to support generation from both images and text prompts.

2. The proposed method employs differentiable rendering that supports perspective projection and occlusion-aware rasterization for 3D curves, distinguishing it from earlier approaches.

3. Experimental results demonstrate that Diff3DS outperforms prior methods in both semantic alignment and user-preferred quality.

4. The writing is clear and the structure of the paper is easy to follow.
summary: The paper introduces Diff3DS, a novel approach for generating view-consistent 3D sketches from either text prompts or image inputs. It employs a differentiable rendering framework that enables the optimization of 3D curve parameters through fully differentiable rasterized images. The method first projects parametric 3D curve control points into 2D using perspective projection. It then applies differentiable rasterization, similar to DiffVG, while correctly handling depth during curve intersections. The framework leverages Score Distillation Sampling (SDS) for supervision, using either a pretrained text-to-image model or an image-guided model. Additional techniques, such as dynamic noise deletion and time annealing schedules, further enhance optimization and generation quality.
Experiments show that Diff3DS generates consistent 3D sketches from both text and image inputs, outperforming previous sketch synthesis methods in terms of CLIP visual similarity and user preference.
weaknesses: Although the paper is well-written and the generated sketches align with the given image and text inputs, there are concerns about the sufficiency of the contribution and the depth of the experimental evaluation:

1. One of the claimed novelty is the use of perspective projection with depth-aware rasterization. However, it seems this approach is not entirely new. A similar perspective projection has already been discussed in 3Doodle (specifically in Appendix A.2), which seems to have no clear difference with the procedure proposed in this paper.

2. While fixing occlusions in 3D curves is a positive addition to the rasterizer, there is no conclusive experimental evidence showing how this improvement impacts the quality of generation. For instance, it remains unclear whether depth-aware rasterization significantly enhances the final results beyond the occlusion analysis in Figure 11.

3. Another novelty is the use of SDS loss supervision, which has not been employed by previous methods. However, applying SDS loss to rasterized images is relatively straightforward and does not present a significant technical difficulty, given its widespread use in NeRF-related tasks. This loss could be easily incorporated into prior approaches without altering their core components. Moreover, the experiments do not convincingly demonstrate the advantage of direct SDS supervision on 3D curves, which I elaborate below.

4. In the image-to-3D sketch task, Diff3DS is compared with 3Doodle and NEF, showing better CLIP scores and improved 3D consistency. However, the fairness of this comparison is questionable, as Diff3DS benefits from multi-view supervision via SDS loss, while 3Doodle and NEF are trained only from a single reference view. Naturally, multi-view supervision leads to better 3D consistency. I think a better comparison would be another baseline like this: first train a 3D representation such as NeRF using the text2image or image2image SDS loss, then use the multi-view renderings of this representation to provide multi-view supervision for training other 3D sketch reconstruction methods, followed by a comparison in terms of quality and efficiency to truly demonstrate the benefits of directly optimizing 3D curves with SDS loss.

Given above points, I believe the paper's contributions may not be substantial enough to justify its acceptance. However, I am open to raise my score if the authors can provide a convincing explaination to the above concerns.

Meta-review: This paper introduces an approach for 3D sketch generation. The core idea is to propose a new differentiable rendering framework for generating view-consistent 3D sketches, which involves perspective projections to render 3D Bezier curves into the 2D image domain. The paper utilized score distillation sampling (SDS) to generate 3D sketches from texts and images. AC confirms that this paper introduces interesting contributions regarding the Bezier curves perspective rasterizer to improve 3D sketch rendering quality, and the successful adaptation to the 3D sketch generation task can shed light on the field.

Decision: Accept (Poster)