Modelling complex vector drawings with stroke-clouds
Authors: Alexander Ashcroft
Ayan Das
Yulia Gryaditskaya
Zhiyu Qu
Yi-Zhe Song
Venue: ICLR 2024 poster (2024)
Keywords: Generative Model
Representation Learning
Sets
Vector Graphics
Primary area: generative models

Abstract: Vector drawings are innately interactive as they preserve creational cues. Despite
this desirable property they remain relatively under explored due to the difficulties
in modeling complex vector drawings. This is in part due to the primarily _sequential and auto-regressive nature_ of existing approaches failing to scale beyond simple
drawings. In this paper, we define generative models over _highly complex_ vector
drawings by first representing them as “stroke-clouds” – _sets_ of arbitrary cardinality comprised of semantically meaningful strokes. The dimensionality of the
strokes is a design choice that allows the model to adapt to a range of complexities.
We learn to encode these _set of strokes_ into compact latent codes by a probabilistic
reconstruction procedure backed by _De-Finetti’s Theorem of Exchangability_. The
parametric generative model is then defined over the latent vectors of the encoded
stroke-clouds. The resulting “Latent stroke-cloud generator (LSG)” thus captures
the distribution of complex vector drawings on an implicit _set space_. We demonstrate the efficacy of our model on complex drawings (a newly created Anime
line-art dataset) through a range
of generative tasks.

Published as a conference paper at ICLR 2024
MODELING COMPLEX VECTOR DRAWINGS WITH STROKE CLOUDS
Alexander Asheroft'
Ayan Das!
Yulia Gryaditskaya"
Zhiyu Qu' Yi-Zhe Song" 1.2
SketchX, CVSSP, University of Surrey, UK SSrrryy Institute for People-Centred AI (PAI), UK
ABSTRACT
Vector representations offer scalability, editability, and storage efficiency, making them indispensable for wide range of digital applications. Yet, generative models for vector drawings remain under-explored, in particular for modeling complex vector drawings. This is in part due to the primarily sequential and auto-regressive nature of existing approaches failing to scale beyond simple drawings. In this paper, we introduce generative model for complex vector drawings, representing them as "stroke clouds" sets of arbitrary cardinality comprised of n-dimensional Bézier curves. Stroke dimensionality is a design choice that allows the model adapt to different levels of sketch complexity. We learn to encode this set of strokes into compact latent codes by probabilistic reconstruction procedure based on De- Finetti's 's Theorem of Exchangeability. A generative model is then defined over the latent vectors of the encoded stroke clouds. Thus, the resulting "Latent stroke cloud generator (LSG)) captures the distribution of complex vector drawings in an implicit ser space. We demonstrate the efficacy of our model in the generation of complex Anime line-art drawings.
INTRODUCTION
Reconstruction
Generation
Figure 1: Our model allows us to perform both probabilistic reconstruction existing samples and generation of new samples matching training data distribution. Reconstruction: We encode vector drawing (left) as a set of Bézier curves and then probabilistically decode it with an MLP- based diffusion model to recreate the drawing (right). Generation: Through the use of latent diffusion model we generate latent codes, and then decode them into vector drawings.
The rise of Diffusion Probabilistic Models (DPMs) (Sohl-Dickstein et al., 2015; Ho al., 2020a) and their spectacular performance on conditional image generation (Rombach et al., 2022; Saharia al., 2022) has prompted the emergence of the subfield of creative visual intelligence. However, diffusion-basee generation targets raster images and leaves drawings, sketches, and other forms of "chirographic data" underrepresented. Vector representations are scalable, editable, and storage ficient properties that are beneficial in digital use cases. Such vector representations, however, due their variable size nature, are mostly (with notable exceptions, e.g. Jain et al. (2023)) incompati- ble with the current image generation pipelines. It is to be noted that modeling chirographic vector modalities have been attempted before (Ha & Eck, 2018; Carlier et al., 2020; Aksan et al., 2020;
The coined by Das al. (2022; 2023)

---

Published as a conference paper at ICLR 2024
Ribeiro et al., 2020; Lopes et al., 2019), but at a smaller scale and complexity. In this paper, attempt to design a generative model directly on vector drawings, specifically some complex ones as depicted in Fig. 1.
Prior attempts at modeling vector drawings were limited to sequential representations (Ha & Eck, 2018; Aksan et al., 2020; Ribeiro et al., 2020). The sequence elements used were discretized 2D points (Ha& & Eck, 2018; Ribeiro et al., 2020), SVG command tokens (Carlier et al., 2020; Lopes al., 2019) and individual strokes (Aksan al., 2020; Das al., 2020b; Chowdhury et al., 2022). Generative models based on sequential representations were demonstrated only for drawings and sketches significantly simpler in complexity than the ones we target in this work. This can be attributed to the inability of sequential models (i.e. RNNs as in Graves (2013), causal Transformers Ribeiro et al. (2020); Aksan al. (2020)) to handle long sequences. A challenge of modeling long- range dependencies is well-known drawback (Pascanu et al., 2013), of such models, even beyond chirographic data, in tasks such as scene synthesis (Paschalidou et al., 2021; Para et al., 2023).
In this paper, we target generative model for vector drawings, specifically focusing complex drawings. To this end, we abandon the sequential approaches. We propose to model highly complex drawings set of its constituent strokes denoting this data structure as "Stroke Cloud". Each stroke can be theoretically represented by any available vector format, such as simple 2D polylines (Ha & Eck (2018)), Bézier curves, differential geometric embeddings (e.g. Aksan et al. (2020)). We here chose Bézier curves for their ubiquitous use in vector art. We then define our genera- tive model on the space of stroke clouds embeddings (sets embeddings), mitigating the long-range dependency issues of sequential representations.
More formally, we define generative model over a X' of arbitrary cardinality. Note that gen- erative models are typically defined on the Euclidean vector space R and do not naturally extend to set spaces, which have element invariance properties. However, the foundation for our set-based generative model has been laid by the recent work of Zaheer et al. (2017), and our solution is sup- ported by De-Finetti's 's Theorem. It follows that we can represent set as a product of independent conditional distributions over the elements, given a latent embedding al., 2019; Zaheer et al., 2017). We can then proceed to build a generative model (o over the set embeddings. We show that such models efficiently scale up to complex drawings and generate plausibly-looking samples.
E(') of the entire set (Lee
Our main contributions can be summarized as follows: (i) We introduce the first generative model for complex vector drawings. Yet, our approach can be used for simpler drawings and any other form of "chirographic data", as demonstrate in the Appendix. (ii) We define our generative model over complex drawings with a novel "stroke cloud" representation, which is a ser of its constituent strokes. To this end, we learn embeddings for each set (stroke cloud) to facilitate the downstream generative model.
The code and the data available at http::/ggithaaccomm/oooodsSssece
2 RELATED WORKS
Sketch Generation First, we focus on the most related recent works for sketch representation and generation. Generating freehand-like sketches from reference images or textual captions remains a challenging task. When generating sketches from images, methods often do not take the abstraction of concepts or objects present in frechand sketches into account, and the produced sketches are some forms of edgemaps (Xie & Tu, 2015; Li et al., 2019; Chan et al., 2022). Alternatively. there is a wide range of sketch generation work based on authentic sketches ranging from creative sketch generation (Ge et al., 2020), shading sketch generation (Li et al., 2020), image-to-skedt to-kethh translation (Liu al., 2020) and face sketch synthesis (Wang al., 2020; Gao et al., 2023). Most of the models are based on a raster sketch representation, which does not reflect the stroke-based nature of authentic drawing that vector sketches do.
To model sketches as a sequence of strokes, sequential and rego-segressive approaches have been used (Ha & Eck, 2018; Zhang et al., 2017). Further, methods for generating vector sketches in- clude SketchHealer (Su et al., 2020), which is a graph-to-sequence approach, SketchPix2Seq (Chen et al., 2017) CNN-based approach for vector sketches, SketchODE (Das al., 2022) neural
2

---

Published as a conference paper at ICLR 2024
ODE based approach and Bézier curve based approach (Das et al., 2020b). More recently denoising probabilistic-base method, SketchKnitter (Wang et al., 2022), has demonstrated state-of- the-art performance in generating simple vector sketches. Despite the impressive performance the requirement to represent all training data with a fixed number of points can prove limiting to wider applications of their method. A recent work (Carlier al., 2020) attempted forcefully impose permutation invariance on stroke-sequences via Hungarian matching (Kuhn, 1955) a cubic time algorithm (i.e. ((n))) that is hard scale beyond low cardinality samples. CLIPascene (Vinker et al., 2022) generates vector sketches from reference images, relying on per-image optimization approach.
Point clouds In this work, rather than representing sketches as sequence of strokes, we represent sketch as a set of (unordered) strokes. This problem bears similarity with point clouds generation and representation. Early point cloud works (Achlioptas et al., 2018; Gadelha et al., 2018) repre- sented point clouds as fixed-sized matrices which enabled existing generative models to be easily applied to the problem. AtlasNet (Groueix et al., 2018) and FoldingNet (Yang et al., 2018) learned mapping from patches in two-dimensional space to three-dimensional space which was able to deform the two-dimensional patches into point cloud shapes. These methods allowed for the gen- eration of point clouds of variable size while also ensuring permutation invariance. An alternative approach is to consider a point cloud as a distribution over direeesiiaaii points. A range of different approaches utilize likelihood- methods for generating point clouds of variable size. Thus, PointFlow (Yang et al., 2019) and DFP-Net (Klokov et al., 2020) utilize normalizing flows to model the distribution of points, while PointGrow (Sun et al., 2020) employs an rugr-segressive model. More recently, (Luo & Hu, 2021) presented a probabilistic diffusion model-based approach, where they condition a diffusion model on a latent representation of the point cloud and proba- bilistically reconstruct point clouds. Colored point cloud generation (Wu et al., 2023) builds on the possible use cases for point clouds as it expands the complexity of the information the point cloud can represent. Our trooke-cloud approach seeks to leverage the flexibility provided by recent point cloud works but focuses on how to model more complex elements than 2D/3D point elements. Namely, we focus how model vector sketches consisting of large number of strokes of diverse shapes.
METHOD
Our generative method consists of two modules: (i) the Stroke cloud Representation Module (SRM), comprised of a Set Transformer Lee et al. (2019) as an encoder and conditional fusion model as the decoder, and (ii) the Latent Stroke cloud Generator (LSG). The SRM module serves as an encoder-decoder, and combined with the LSG module it allows us to generate new drawings representative of the training dataset. The latent code generated by the LSG decoded by the SRM into N individual strokes, where N is a hyperparameter. We are therefore able to probabilistically reconstruct complex drawings with a variable number strokes.
dif-
3.1 DRAWING REPRESENTATION
We represent a drawing in our dataset D as a set of strokes S = (s((),s(()) (2,,),))))))))))) s(3) :... s E D. Note that the cardinality, N, of set (the number of strokes line drawing) varies across drawings in the training data.
We represent each stroke in the drawing as a Bézier curve. Unless specified otherwise, we use quadratic Bézier curves, represented as follows: s(1) (I1, (I1,y1, 01, 1/2 33, v3), where each pair of (I;, 31) are the coordinates of the i-th control point. While these relatively simple strokes lack the complexity of authentic hand-drawn strokes they can be used to represent complex drawings. For more information on stroke design and usage of Bézier curves of higher degrees please refer to the Appendix E.
3.2 SET REPRESENTATION MODULE
We model our Set Representation Module (SRM) as a generative conditional model. As a generative model, we use Denoising Diffusion Probabilistic Model (DDPM) (Ho et al., 2020a). The training
3

---

Published as a conference paper at ICLR 2024
objective can be formulated follows: máx(w) ESED log (ogp(,)(S),
where v are trainable parameters.
Stroke cloud joint probability distribution is challenging to define generative model directly on the set space S due to the varying number of strokes in each line drawing. A solution that explore here is to first learn to transform set into a latent representation. Our solution is inspired by the work by Zaheer et al. (2017) and De-Finetti's Theorem of Exchangeability" stating that set (exchangeable sequences of random variables) can be modeled as a probability distribution over its constituent elements given some latent representation of the set.
Therefore, we decompose drawing S into conditionally independent parametric density functions of the individual strokes s E S, conditioned on latent embedding of drawing S:
P( (S) Ilees ps(2/ Ep)S)
where (S) is the drawing encoder into latent space with learnable parameters Φ.
Now, we can consider strokes as I.I.D, and can rewrite the training objective in Eq. (1) as follows:
max (b ESED log s( (S) = (a,0) ESED Eses log po (s/z zz (S))
(3)
Training As an approximation to the true log-likelihood in Eq. (3), we optimize a noise-estimator model trained on noisy versions of strokes S at diffusion timestep (noise level) t:
min ESED [ESSS, e~,(), (u(T)) |Iea (st, t z = (S)) - <1 (0,0)
(4)
where ea(-) ) is the conditional noise-estimator: A noisy stroke, 8r, is obtained as follows: St vais + vI are for all timesteps [1.7], where a E [0. 1] is a monotonically decreasing diffusion schedule and R6x11 The noise is added each control point of stroke. For more detail on the standard Diffusion Model formulation, please refer to appendix F.
are
The stroke cloud encoder & The stroke cloud encoder is an important element of the SRM, as enables the representation of strokes as independent random variables. Due to its theoretically guar- anteed permutation-invariat (PMA) (with one seed vector) proposed by Lee et al. (2019) in order to encode a given set S into compact latent code z.
nature, we use Set Transformer with Pooling by Multihead Attention
Reconstructing the stroke cloud Given a trained noise estimator with the parameters 0" and a trained stroke cloud encoder such that z = Eg- (S), can decode the set by running any diffusion sampler:
f SAMPLER(CO- 2) | [1.
(5)
where SAMPLER(() is any sampling procedure compatible with DDPM training (Ho et al., 2020a). We discuss the choice of the sampler in more detail in Sec. 4.
Note that since we assume the strokes to be independent and identically distributed random vari- ables, the model not aware of the set's cardinality N, which is the number of strokes in a sketch. However, due to the presence of the expectation Eyes in Eq. (3), the model does implicitly encode the relative importance of each stroke. We treat the cardinality N of the reconstructed set S as hy- perparameter, and discuss it in more detail in Sec. 4. In Sec. 4, we show that both the hyperparameter N and the exact sampling procedure influence the visual quality of the reconstructed drawing.
3.3 LATENT STROKF CLOUD GENER ATOR
To enable unconditional generation, we leverage a latent generative model that we term "Latent Stroke cloud Generator" (LSGG). To train the LSG model, we extract embeddings of the drawings in our dataset DD
D 2 z= Eg- E (8) (S) S SED)
(6)
We provide detail on De-Finetti's Theorem of Exchangcability in the Appendix G.
4

---

Published as a conference paper at ICLR 2024
%x:
Xx-1
Figure 2: An illustration of the reverse diffusion process for set of 1000 strokes with a DDIM sampling method. The original drawing was comprised of 350 strokes Repeated strokes are 're- drawn' on top of one another.
LSG is then simple generative model defined over the latent vectors z: pa( with trainable pa- rameters qb. Just like Sec. 3.2, we realize py(e using a diffusion model. Specifically, we train a parametric noise estimator Ev on the noisy latents 2, = vanz /1 agy. This estimator estimates the noise component e:
min EEDD e-N(0..), (u(1)) |lap (zc, t) <11]
EXPERIMENTS & RESULTS
4.1 DATASET PREPARATION
A key challenge in generative modeling for complex vector drawings is acquiring a sufficiently large dataset. Due the unavailability of such datasets, much of chirographic modeling (Ha & Eck, 2018; Aksan et al., 2020; Das et al., 2023; Wang et al., 2022; Das et al., 2022) has focused on the QuickDraw dataset, which contains a vast number of very simple drawings. For reference, the average number of strokes in sketches from QuickDraw is 5.
To demonstrate the effectiveness our stroke cloud-based sketch generation framework in generat- ing complex vector sketches, synthetically generate dataset that name Anime- Anime-Vec/l Vec 1Ok, derived from the Danbooru2019 dataset (Branwen et al., 2019) of anime raster images. We then use this dataset to train our model.
The Danbooru2019 image database comprises 3.69 million anime-style artworks in raster format, along with over 106 million annotations. To create our dataset, randomly select 10,000 samples from a subset of Danbooru 2019 Portraits, which are portraits cropped from the original Danbooru dataset. We then transform these samples into line drawings using style-transfer Generative Ad- versarial Network (GAN) as described in Chan et al. (2022). Finally, we utilize a line art vectorizer by Mo al. (2021) to convert these synthetic line drawings into complex vector sketches, consisting of quadratic B' czier curves. This process is illustrated in Fig. 3. For more details, please refer to Appendix B. On average, the sketches our dataset consist of 305 strokes.
Original
Sketch
Vector
Stroke Cloud
Figure 3: To generate the Anime- Vec 10k dataset we take an original image from the Danbooru 2019 Portrait dataset and use a style GAN to convert 'sketch style'. We then apply a vectorizer to generate a set of quadratic Bezier curves.
5

---

Published as a conference paper at ICLR 2024
Sampler and stroke number
DDIM 100
DDIM 500
DDIM 1000 DDIM 5000
DDPM 1000
FID
191
34
9.8
11.3
Table Quantitative comparison of drawings generated under different sampling conditions. DDIM sampling was done with 30 steps while the DDPM sampling was done with 1000 steps.
4.2 STROKES REPRESENTATION AND EMBEDDING
As we introduced Sec. 3.1, our framework, each stroke drawing is represented as g(i) (I1, 22, V/, F3, ys), where each pair of 2,, Yi denotes the control points of a quadratic Bézier curve. Using a Set Transformer (Lee et al., 2019), we encode each drawing as a set of variable cardinality, with each stroke
element in the set. The resulting latent code conditions MI.P- based diffusion model, which learns to generate elements of the set.
To address spectral bias in the MLP (Tancik et al., 2020), employ sinusoidal embedding for each control point coordinate. Increasing the dimensionality of the sinusoidal embedding helps to mitigate the spectral bias, can be seen Fig. 4.
Figure 4: Varying the dimensionality of the sinusoidal positional embedding can have a significant impact the drawing quality. While the general position of each stroke in the low-dimensional embedding drawings is correct they lack the fineggrained accuracy of the drawings with the higher- dimensional embedding.
4.3 STROKES SAMPLING
The variability the number strokes within drawings poses challenges in vector drawing gener- ation. While existing methods often employ autoregressive techniques fix the number of strokes or polyline points, our approach utilizes a more flexible probabilistic reconstruction process. This approach allows to learn compact latent representations while effectively reconstructing these latent codes as observable strokes.
After training, can condition the SRM with given latent code generate samples and attempt to reconstruct the encoded drawing. However, the complexity of the drawing is generally unknown, in- troducing the challenges of both over-sampling and under-sampling. For more detailed information on probabilistic reconstruction, please refer to Appendix D.
Over-sampling: When we generate significantly larger number of samples than the original num- ber of strokes in the drawing, over-sampling can occur. This is illustrated in the leftmost drawing of Fig. The generative process may result in particular strokes being sampled more frequently, leading to slight variations and noise in some sections of the drawing. Overall, the drawing quality remains largely unchanged, with most strokes being 'redrawn' on top of one another.
Under-sampling: On the other hand, under-sampling involves generating too few strokes, result- ing in a sparsely populated canvas, as seen in the right-hand drawings of Fig. Under-sampling significantly impacts the quality of the drawing. Table shows the effect of varying the number of generated strokes on the visual quality of the drawing as measured by the FID. These results confirm that the effect of under-sampling the visual quality is significantly more severe than over-sampling.
Sampler: The choice of sampling method can also influence the quality and style of the drawing. Fig. demonstrates the effects of varying the sampling method. Our reconstructive method may re-sample some strokes multiple times, affecting the variance in each sample and, consequently, the

---

Published as a conference paper at ICLR 2024
700
400
100
Figure 5: The number of generated samples has significant impact the generated drawing, With only 100 samples (right) the drawing is sparsely populated by strokes and key features are missing. However, with 1000 samples (left) more accurate reconstruction is achieved
DDIM
DDPM
DDPM 1
DDPM
DDPM
Figure 6: The stochasticity of the sampling method has significant impact on the quality of the drawings. We utilize a DDIM sampler (left) which the most resilient to the re-sampling problem. We increase the stochasticity of the sampling process by using DDPM sampler and multiplying the variance by scale factor (Das et al., 2023). The scale factor is displayed beneath each drawing.
drawing's appearance. Using deterministic DDIM sampler produces drawings with clean edges and minimally noticeable re-sampling. The small variance in re-sampled strokes is concealed by the line's thickness. However, using a stochastic sampling method creates shading effect similar to artist sketching composition. To control this effect, adjust the variance in the de- de-noising step using a scale factor (Das et al., 2023), increasing or decreasing the stochasticity of the process. Fig. 6 illustrates the impact of this scale factor the drawing's style.
4.4 GENERATION
To generate a drawing we must first generate latent code with the LSG, this is done using DDIM sampler and 30 time steps. We decode the resultant latent vector with the SRM, with a DDIM sampler and 30 time steps, into a drawing comprised of 1000 strokes. Drawings generated by our model are shown in Fig. 7. To generate sketches in this figure, generate the number of strokes much larger than the average number of strokes the training dataset. While selecting the correct number of samples is important choice in generating complete and highly complex drawings, if drawing is incomplete we do have the option of appending more generated samples using the same latent code.

---

Published as a conference paper at ICLR 2024
Figure 7: Drawings of 1000 strokes created by decoding latent vectors generated by the LSG with the SRM
4.5 INTERPOLATION
The SRM plays crucial role in reconstructing drawings based on a given latent condition. There- fore, it is essential to assess the model's robustness to different conditions. On the other hand, the LSG's primary function is to generate a latent code that the SRM can decode successfully.
SRM: The SRM does not operate unconditionally; instead, it relies on being conditioned by latent stroke cloud to perform the reconstruction. Consequently, it needs to be resilient to variations in codes generated by the LSG. As illustrated in Fig. 8, when we interpolate between two encoded stroke clouds, the SRM retains semantic features from each stroke cloud, even for conditions it has not encountered previously.
LSG: The LSG serves as the core generative component of our model, providing the essential latent code for the SRM to work with. Fig. 9 demonstrates that it is possible to interpolate between two
8

---

Published as a conference paper at ICLR 2024
Reconstructed sketch
Reconstructed sketch
Sketches from the interpolated latents
Figure 8: Interpolating between encoded drawings leads to gradual morphing between drawings. While noisy in the middle of the transition, clear features are still distinguishable.
Sketch from randomly sampled latent
Sketch from candonily sampled
Sketches from the interpolated latents
Figure 9: Two random vectors were generated and interpolated between. Each vector was used as the initial vector the LSG denoising process.
randomly selected noisy vectors, resulting in distinct, drawings. Furthermore, we show in Fig. 10 the impact of gradually adding noise to the originally selected random vector. The sketches obtained from the noised latent vectors are also distinct and plausible.
Sketch from randomly sampled latent
Naise addied the latent: once
Noise arided latent: twice
Naise added latent: times
Noise added the latent: 4 times
Naise addied latent: times
Figure 10: A random latent vector was sampled, and then Gaussian noise was added gradually to obtain a new latent. The amount of noise added increases from left right.
DISCUSSION
In this paper, we propose modeling complex vector drawings of structurally complex ele- ments. We learn to embed these drawings into compact latent codes. These latent codes then con- dition an MLP--ased diffusion model that enables the efficient generation of highly complex vector drawings through latent diffusion process, supported by De-Finetti's Theorem of Exchangeability. One limitation our approach the unknown a priori number of strokes to sample. However, we have shown that oversampling produces visually pleasing sketches in which some strokes overlap. Such strokes can be potentially removed in post-processing by amalyzing the areas of overlaps. Lim- ited by the lack of datasets of complex vector drawings, trained synthetic data. However, the strokes produced by the automatic vectorizer are shorter than those hand-drawn sketches. In the complex
provide additional results, showing how method support strokes by increasing the number of control points in our stroke representation. Moreover, we also show in the supplementary how additional attributes such stroke width can be supported by our framework. In summary, we proposed the first approach to model complex vector drawing in generative context. Our code and the data are available at https://githubb.c. Sttps/thuwbbooo/oooo/otttroo.o Cloud.
supplementary,

---

Published as a conference paper at ICLR 2024
REFERENCES
Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, and Leonidas Guibas. Learning representa- tions and generative models for 3d point clouds. In International conference on machine learning. pp. 40-49. PMLR, 2018.
Emre Aksan, Thomas Deselaers, Andrea Tagliasacchi, and Otmar Hilliges. Cose: Compositional stroke embeddings. NeurlPS, 2020.
Gwern Branwen, Danbooru community, and Anonymous. Danbooru2019: A large-scale crowd- sourced and tagged anime illustration dataset, 2019.
Alexandre Carlier, Martin Danelljan, Alexandre Alahi, and Radu Timofte. Deepsvg: A hierarchical generative network for vector graphics animation. NeurlPS, 2020.
Caroline Chan, Frédo Durand, and Phillip Isola. Learning to generate line drawings that convey geometry and semantics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7915-7925, 2022.
Yajing Chen, Shikui Tu, Yuqi Yi, and Lei Xu. Sketch-pix2seq: a model to generate sketches of multiple categories. arXiv preprint vrXiv:1709.04120 2017.
Pinaki Nath Chowdhury, Aneeshan Sain, Ayan Kumar Bhunia, Tao Xiang, Yulia Gryaditskaya, and Yi-Zhe Song. Fs-coco: Towards understanding of freehand sketches of common objects in con- text. In European Conference on Computer Vision, pp. 253-270. Springer, 2022.
Ayan Das, Yongxin Yang, Timothy Hospedales, Tao Xiang, and Yi-Zhe Song. Béziersketch: A generative model for scalable vector sketches. In Computer Vision-ECCV 2020: 16th Euro- pean Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXVI 16, pp. 632-647. Springer, 2020a.
Ayan Das, Yongxin Yang, Timothy Hospedales, Tao Xiang, and Yi-Zhe Song. Béziersketch: A generative model for scalable vector sketches. In ECCV, 2020b.
Ayan Das, Yongxin Yang, Timothy M Hospedales, Tao Xiang, and Yi-Zhe Song. Sketchode: Learn- ing neural sketch representation continuous time. In Tenth International Conference on Learn- ing Representations 2022, 2022.
Ayan Das, Yongxin Yang, Timothy Hospedales, Tao Xiang, and Yi-Zhe Song. Chirodiff: Modelling chirographic data with diffusion models. In The Eleventh International Conference on Learning Representations, 2023.
Matheus Gadelha, Rui Wang, and Subhransu Maji. Multiresolution tree networks for 3d point cloud processing. In Proceedings of the European Conference Computer Vision (ECCV), pp. 103- 118, 2018.
Fei Gao, Yifan Zhu, Chang Jiang, and Nannan Wang. flmann-nnspired facial sketch synthesis with dynamic adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7237-7247, 2023.
songwei Ge, Vedanuj Goswami, Lawrence Zitnick, and Devi Parikh. Creative sketch generation. arXiv preprint arXiv:2011.10039 2020.
Alex Graves. Generating sequences with recurrent neural networks. CoRR, abs/130s (303338880000 2013.
Thibault Groueix, Matthew Fisher, Vladimir G Kim, Bryan C Russell, and Mathieu Aubry. A papier-måché approach learning 3d surface generation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 216-224, 2018.
David Ha and Douglas Eck. A neural representation of sketch drawings. In ICLR, 2018.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS, 2020a.
10

---

Published as a conference paper at ICLR 2024
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-653 6851, 2020b.
Ajay Jain, Amber Xie, and Pieter Abbeel. Vectorfusion: Text-to-svg by abstracting pixel-based diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1911-19200 2023.
Roman Klokov, Edmond Boyer, and Jakob Verbeek. Discrete point flow networks for efficient point cloud generation. European Conference on Computer Vision, pp. 694-710. Springer, 2020.
Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):33-77, 1955.
Juho Lee, Yoonho Lee, Jungtack Kim, Adam R. Kosiorek, Seungjin Choi, and Yee Whye Teh. Set transformer: A framework for attention-based cormuutationnnnvariin neural networks. In JCMI,, 2019.
Mengtian Li, Zhe Lin, Radomir Mech, Ersin Yumer, and Deva Ramanan. Photo-sketching: Inferring contour drawings from images. In 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1412. IEEE, 2019.
SuChang Li, Kan Li, Ilyes Kacher, Yuichiro Taira, Bungo Yanatori, and Imari Sato. Artpdgan: Creating artistic pencil drawing with key map using generative adversarial networks. In Compu- futional Science-ICCS 2020: 20th International Conference, Amsterdam, The Netherlands, June 3-5, 2020, Proceedings, Part VII 20, pp. 285-298. Springer, 2020.
Runtao Liu, Qian Yu, and Stella X Yu. Unsupervised sketch photo synthesis. In Computer Vision- ECCV 2020: 16th European Conference, Glasgow, UK, Augus! 23-28, 2020, Proceedings, Part III 16, pp. 36-52. Springer, 2020.
Raphael Gontijo Lopes, David Ha, Douglas Eck, and Jonathon Shlens. learned representation for scalable vector graphics. In ICCV, 2019.
Shitong Luo and Wei Hu. Diffusion probabilistic models for 3d point cloud generation. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2837-2845, 2021.
Haoran Mo. Edgar Simo-Serra, Chengying Gao, Changqing Zou, and Ruomei Wang. General virtual sketching framework for vector line art. ACM Transactions on Graphics (TOG), 40(4):1-14, 2021.
Wamiq Reyaz Para, Paul Guerrero, Niloy Mitra, and Peter Wonka. Cofs: Controllable furniture layout synthesis. ACM SIGGRAPH 2023 Conference Proceedings, pp. 1-11, 2023.
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In International conference on machine learning, pp. 1310-1318. Pmlr, 2013.
Despoina Paschalidou, Amlan Kar, Maria Shugrina, Karsten Kreis, Andreas Geiger, and Sanja Fi- dler. Atiss: Autoregressive transformers for indoor scene synthesis. Advances in Neural Informa- fion Processing Systems, 12111222102000 2021.
Leo Sampaio Ferraz Ribeiro, Tu Bui, John Collomosse, and Moacir Ponti. Transformer-based representation for sketched structure. In CVPR, 2020.
Sketchformer:
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Informa- tion Processing Systems, 2022.
Jascha Dickksics,,,,, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In ICMI. 2015.
11

---

Published as a conference paper at ICLR 2024
Guoyao Su, Yonggang Qi, Kaiyue Pang, Jie Yang, and Yi-Zhe Song. Sketchhealer graph-to- sequence network for recreating partial human sketches. In Proceedings of The 31st British Ma- chine Vision Virtall Conference (BMVC 2020), pp. 1-14. British Machine Vision Association, 2020.
Yongbin Sun, Yue Wang, Ziwei Liu, Joshua Siegel, and Sanjay Sarma. Pointgrow: Autoregres- sively learned point cloud generation with self-attention. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 61-70, 2020.
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara -ridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Process- ing Systems, 33:7537-7547, 2020.
Yael Vinker, Yuval Alaluf, Daniel Cohen-Or, and Ariel Shamir. Clipascene: Scene sketching with different types and levels of abstraction. arXiv preprint arXiv:2211. 17256, 2022.
Qiang Wang. Haoge Deng, Yonggang Qi, Da Li, and Yi-Zhe Song. Sketchknitter: Vectorized sketch generation with diffusion models. In The Eleventh International Conference on Learning Repre- sentations, 2022.
Tianying Wang, Wei Qi Toh, Hao Zhang, Xiuchao Sui, Shaohua Li, Yong Liu, and Wei Jing. Robocodraw: Robotic avatar drawing with gan-based style transfer and time-efficient path op- timization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp- 10402-10409, 2020.
Zijie Wu, Yaonan Wang, Mingtao Feng, He Xie, and Ajmal Mian. Sketch and text guided diffusion model for colored point cloud generation. arXiv preprint arxiv:2308.02874 2023.
Saining Xie and Zhuowen Tu. Holistically-nested edge detection. In Proceedings of the IEEE international conference on computer vision, pp. 1395-1403, 2015.
Guandao Yang, Xun Huang. Zekun Hao, Ming- Yu Liu, Serge Belongie, and Bharath Hariharan. Pointflow: 3d point cloud generation with continuous normalizing flows. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 4541 2019.
Yaoqing Yang, Chen Feng, Yiru Shen, and Dong Tian. Foldingnet: Point cloud auto-encoder via deep grid deformation. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 206
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabás Póczos, Ruslan Salakhutdinov, and Alexander J. Smola. Deep sets. In NeurlPS, 2017.
Yu-Yao Zhang, Fei Yin, Yan-Ming Zhang, Cheng-Lin Liu, and Yoshua Bengio. Drawing and rec- ognizing chinese characters with recurrent neural network. IEEE transactions on pattern analysis and machine intelligence, 40(4):849-862, 2017.
12

---

Published as a conference paper at ICLR 2024
A IMPLEMENTATION DETAILS
Network. The SRM network contains Set Transformer Lee al. (2019) encoder and an MLP-based diffusion model as a decoder. The network architecture can be seen Fig. 11. Our network was trained end-to-end before the LSG was trained on the resulting latent codes.
Anime-Vedd 10k. 10k subset the Anime- Vee 10k data set was used for training. The model was trained for 72 hours
single RTX 4090 with batch size of 128 and initial learning rate of le-4 that was decayed to 5e-5. After the initial training period, we applied KI. annealing for another 24 hours, increasing the KL scale factor from 0 to 1e-8. We trained our model with linear noise schedule of Bmin le 4, Smar le and 200 time steps.
LSG. Each LSG was trained the latent data obtained by passing the training data through the trained encoder. The LSG was then trained for 12 hours with a batch size of 2048 and an initial learning rate of le-4 that was decayed 5e-5. The LSG was trained on a non-conditional version of our MLP. We used a scaled-liness noise schedule of 8mn 2e 2, fmaz = le 4 and 4000 time steps.
LSG
Tainformet
D
D
MLP
D
Encoder
Decoder
Time embedding
Figure 11: Network diagram of the SRM and LSG for training.
B DATASET
Danbooru2019 (Branwen et al., 2019) is a scraping based dataset from the image board Danbooru. Images scraped and then filtered according user-generated tags and then subsets of this data made. The Danbooru2019 dataset has permission from host of the Danbooru website, copyrighted material may have been uploaded by users. Furthermore, the user-generated tags used to filter the images may not be entirely accurate resulting in unwanted images in the final dataset. For our Anime-Vec108 1kc dataset, we manually reviewed all images and removed erroneous content help address this issue. The examples of training data are shown in Fig. 12.
C SAMPLES
In this section, we show more samples from the unconditional generation of the LSG for the Anime- Vec10k model shown in Fig. 14.
Additional interpolation results from the SRM are shown Fig. 13. In this case, we linearly inter- polated between encoded drawings and then used these latent vectors condition the SRM. Despite
13

---

Published as a conference paper at ICLR 2024
Figure 12: Examples of the training data.
training the SRM with regularization the model is unable to handle conditions that differ greatly from the training data. We consider this to be a limitation of the limited-size dataset.
Figure 13: Interpolating between encoded drawings leads to a gradual morphing between drawings.
While noisy in the middle of the transition clear features are still distinguishable.
D RECONSTRUCTION
This section extends Sec. 4.3 of the main paper. With the probabilistic reconstruction of the drawing, is unlikely to guess in advance on the exact number strokes required to accurately model it. Instead, it is much more likely that there will be either too many too few strokes. Fig. 5 showed that selecting too few elements is worse than too many, in addition, Fig. 15 shows that simpler sketches if too few strokes are generated the incompleteness be more pronounced. When an
14

---

Published as a conference paper at ICLR 2024
Figure 14: Additional samples generated by our model trained on the Anime-Vecl Voc10k dataset.
element is sampled several times there is some variation in that element as was shown in Fig. 6. However, the thickness of each line in the drawing can mask this to make it look like there is no variation present at all. We show more clearly in Fig. 16 that even with a DDIM sampler there is some variation in repeated elements even if they are very similar.
E STROKE DESIGN
While our method can generate large sets of variable cardinality the visual quality of the results also depends on the complexity and quality of each element in the set: the stroke. We consider two primary limitations in the stroke quality, the availability of training data and the complexity of the stroke the SRM can generate. The line art vectorizer (Mo et al., 2021) formed the backbone of our dataset preparation pipeline. As it produces short line segments in the form of quadratic Bézier curves, we chose quadratic Bézier curves for our experiments in the main document.
However, the general form of each element of the set could be written as follows: s(1) (I), yl, 22, V2, In. Yn, m,), where the control points specify the structural complexity of
Control points
Meta
15

---

Published as a conference paper at ICLR 2024
21
Figure 15: 10 Stroke drawings of QuickDraw cats. The incompleteness of the drawing is quite obvious in simpler sketches.
Figure 16: 1000 stroke drawings sampled with DDIM. With thinner rendered lines it is easier to see the variation in stroke placement. Repeatedly sampled strokes are sampled directly on top of one another but close enough that with thicker lines it is much harder to see.
each stroke, and the Meta-data can include rendering information. Two examples of different stroke designs to what was used in the majority of the experiments can be seen in Figs. 17 and 18.
First, we trained model on subset the Anime 10k data set but using variable line thickness. The line thickness is provided by the vectorizer we used to generate the dataset (Mo et al., 2021). Each stroke the set 7-dimensional vector ness. This serves generative framework, We show the representative, generated by our model, sketches with varying thickness strokes in Fig. 17.
(æ1, //. V2, F3. y3, I), where is the line thick- example of how additional metadata can be included in our vector-sketch
Then, we experiment with a dataset of hand-drawn sketches - the QuickDraw dataset. The strokes in this dataset are stored polylines, consisting of multiple points. We represent cats from the QuickDraw with 15 control point Bezier curves (Das et al., 2020a) and then represent each drawing set of these strokes. We then trained our model as we did for the Anime dataset, with the only difference that now we use 15 control point Bezier curves rather than quadratic Bezier curves. In Fig. 18, show samples generated with our method.
16

---

Published as a conference paper at ICLR 2024
Both cases show that, while we used simple stroke design (a quadratic Bézier curve with contract stroke width) for the majority of the experiments, our framework allows us to support more complex strokes and additional stroke attributes.
Figure 17: 1000 stroke samples generated with variable line thickness
For each sketch example, 50 stroke samples were generated with our method, after training the model on the QuickDraw dataset cats category. Each stroke was represented by 15 control point Bézier curve.
Figure 18:
F DIFFUSION MODELS BACKGROUND
To learn the probability distribution over data x. Diffusion models corrupt training data by slowly injecting noise and then learn to reverse the corruption, such that the obtained models can gradually transform random noise into samples for data generation.
Forward Process
For each training sample .r Idato (Fo), discrete Markov chain .i,..., ZT is formed by the forward process (also known as diffusion process). This process is defined as Markov chain which slowly adds Gaussian noise to the data according to variance schedule 31,... 8T
T
((trT())) = II gg(t
(8)
17

---

Published as a conference paper at ICLR 2024
q(ze|==_1) = (xx /1 1.11)
(9)
we know ((=))1122)) we could sample data from the data distribution ((m)) by first sampling TT from ((xT) (isotropic Gaussian) and then sampling from g(zz-1(zz) until we get rn. However, is difficult to estimate g( (( since it needs the entire dataset to do so. Therefore, po is proposed approximate the conditional probabilitiess (( during the reverse process.
Reverse Process In the reverse process, diffusion models have to denoise the perturbed data (starting random noise p(æT) NV (2T; O)) back the origin data zo- Mathematically, diffusion models are defined as
pe(xo) po (1o::))d
(10)
for which the joint probability distribution ps( defines the reverse process
po(Xo:T) = -((x)) II t=1
po (x, (x,-1/x,)
(11)
pe xx [xx) (x uo (xe, 1), Eg (x,, 8))
(12)
The training object is then optimize the variational bound on the negative log-likelihood as:
log pe (xo) S log po (xo) + (wL(g(X1:r))))) IIpo (x1:T (xo)
log Po (xo) + xxxxr>xxxxxxx log 9(x1-T X
(Xo.T) /o (x0)
(xx1:)) (x0) log po (x
log (xu) + Egg
log
po (x0:2)
(13)
= -Fa[[ log g(x1:11 [x0)
px(x).T)
loog 1 (x1:T|100) po (xxx))
Let TVLB Exp(x
-By(xo) log po (Xp)
This is the equivalent of:
LVLB LT + 1..1....4 Lo
where Lta DKL. (1(xT ((xx|xx) (x0) po())))
(14)
L. DKL Xx (x(11, (x(x)(Xx41)) forl StST-
Lo log pe (1)
The training objective for Eq 12 is to get po( l), while not involving Eo (), it is set time-dependent constants o²=1. Furthermore, instead of predicting [1( t) by neural network, Ho et al. (2020b) proposed to utilize an approximator co ca(,,, t) to predict noise e from 24, which is proven to be more effective than optimizing (z)xttt) The simplified training objective is:
Liimpte (0) (e |||e en (x 1)1
(15)
Sampling Once trained, the network estimates noise e from the sample a, at timestep t (co(ze, t)). (z), t) can then be derived from eø co(x,, by the following:
1 a₁ co (x),
(xt
(16)
vat
where a, = 1 3, and a = II.=1 a, We can then sample data from po 1 according to Eq 12 until we reach 80
18

---

Published as a conference paper at ICLR 2024
G DE FINETTI'S THEOREM
Exchangeability
For given set (X,)== of objects let Xx denote the joint probability. This set is exchangeable [x,,..,x = "000) i.e. if for every permutation π {1,...,? n2 + (1,,.,,) n).
De Finetti General Representation Theorem
X1, X2,..., X, is an exchangeable sequence of variable length with probability measure P. then there exists distribution function Q on F, the set of all distribution functions on R, such that the joint distribution of (X1, X2, Xn) has the form:
X2,, / II F(X))(((F)
Xn)
(17)
==1
where an unobservable distribution function.
Q(F) lim (a(F)) 12000
(18)
the probability measure on the space of functions F, defined as limiting measure n - oc on the empirical distribution function F
19


Reviews:
[{"review_id": "UOppnhceGF", "reviewer": "Reviewer_DN8s", "summary": "The paper purports to present a method for generating _vector representation of complex sketches_.\nThe problem is important and has the following challenges:\n1. most work on generative modeling focuses on images, so vector representation is underrepresented.\n2. the number of _strokes_ within each sketch is variable.\n3. autoregressive methods using a sequence representation fail to scale up.\n4. the sequence representation is sensitive to the ordering of strokes.\n\n---\n\nThe claimed contributions are:\n1. proposing the first method for generating _highly complex drawings_.\n2. using a _set representation_ instead of a sequence representation.\n\n---\n\nThe paper:  \n1. (MAJOR) argues that a set representation is is theoretically possible because  of  `De Finetti General Representation Theorem` which states that for an `exchangeable` set with a pdf for each of it's constituent sets (one realization of the permutation), there exists a pdf that describes all the permutations in the limit.\n2. (MAJOR) presents a practical way to construct this permutation-invariant pdf, so that we can sample sketches from this distribution tractably.\n3. (MINOR) presents a new dataset called `Anime-Vec10k` which which is significanlty more complex than existing datasets (notably QuickDraw!)\n\n---\n\nThe method used to achieve the goal of  \"generating _vector representation of complex sketches_\" is roughly the following:\n1. Consolidate image into sets of bezier-curves.\n2. Use a Set-Encoder to encode the stroke-set to get a latent.\n3. Use a VAE loss to regularize the space of latents.\n4. Use an MLP based conditioning denoising network to get from (500/1000) _random_ strokes to the actual strokes.\n5. The important point in point 4 is that **each stroke is considered iid**.", "questions": "I already asked most questions.\n\n1. Figure 5: Why does the DDPM sampler seem to have more strokes than the DDIM sampler? or are the strokes just placed closer together in the DDIM sample?\n\n2. Do you plan to release the dataset?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "**The problem setting is valuable to the community.** \n\n---\n\nWhile diffusion models have been providing very high quality generative models in the raster world, they have not made as much of a splash in the vector world. So I commend the authors for tackling this problem.\n\n**Very good exposition.**\n\n---\n\nThe paper is very easy to follow. While the abstract could possibly use some more technical details. Everything from the introduction on flows logically. The introduction is superb, even though it spends a bit more time describing why `creational` representations matter than I would have liked. The introduction still sets up the problem well - the problem with sequences (ordering and non-scalability of the autoregressive approach) and the problems with using a set approach (variable and unknown cardinality). The related work section is quite comprehensive to me. I would still like to suggest two references which might be interesting to the reader in the weaknesses section.\n\n**The core idea is simple to implement**\n\n---\n\nThe whole paper is based on simple building blocks, so any (re-)implementation should be easy.", "weaknesses": "While I like the paper, in its current form the paper has many weaknesses:\n\n**(MAJOR): Sampling issues**:\n\n---\n\nThe authors clearly point out that the number of strokes is an issue, but simply brush it away by saying we use 1000 strokes.\nThis is a glaring flaw. What if the sketch is simpler? What happens to the other strokes? Are they duplicated? \n\nThe discussion in Appendix E is is simply not enough to provide a solution.\n\n**(MAJOR): Matching issues/ iid assumption of strokes **:\n\n---\n\nRelated to sampling is the issue of matching. In the diffusion model equation (3), you simply decompose the sketch into iid strokes. This is a **VERY** strong assumption. I do not see how during sampling, one cannot get a degenerate solution of just repetetively denoising into the same stroke. There is no question asked about how the iid sampling affects the proposed pipeline.\n\nIn the same way, how is the sequence $\\mathbf{s}$ generated at training time? Do you let the randomness of the random strokes take care of matching the proper final stroke? Why is there no hungarian matching? Assuming normalized coordinates in $[-1, 1] \\times [1, 1]$ , how does it make sense for the denoising network to take a random stroke at (-1, -1) and try to denoise it to (1,1), instead of just recognizing (with hungarian matching) or simple euclidean distance that it is much easier to denoise the stroke closer to the final location?\n\n**(MAJOR):  Sequence representation/Sequence lengths**\n\n---\n\nIt is not clear at a first glance (unless I am wrong) that the paper does the following:\n1. Takes in a **variable** number of input strokes\n2. Generates (denoises) a **fixed** (1000) number of output strokes.\n\nThe motivating factor of the paper, and the way the paper is currently written suggest that the model is capable of generating a variable number of strokes. It seems that is not possible\n\n**(MAJOR) No quantitative results**\n\n---\n\nI said previously that the paper has missing citations. I will mention the actual links later but describe them here:\n\nThere has been recent work (<1.5years) in the autoregressive generation area. The work is based on how to make generation permutation invariant which is exactly the problem being tackled here. There are two papers in this area: [Paschalidou] and [Para]. [Paschalidou] use a learnable query vector that looks at the previously generated sequence and predicts the next one. [Para] uses a set encoder and a sequence decoder with mask tokens to perform controled generation.\n\nThe authors do not cite these papers. The authors have no baselines as they (rightfully) claim that previous work does not scale - but there [Paschalidou] has code available and it would be easily adaptable to their current training regime\n\n[code](https://github.com/nv-tlabs/atiss)\n\nAll you have to do is make each token the sum of its control point embeddings! And then introduce some form of conditioning - either a single condition token or an encoder as done in [Para].\n\nThis should significantly strengthen the paper - we see exactly how slow and underwhelming the other methods are, how slow to sample, what the FID is, and qualitative results as well. I would really like to see those result\n\n**(MINOR)  Few qualitative results**\n\n---\n\nWhile there are a decent number of qualitative results in the paper already - the dataset itslef is small - 10k samples. I would request the authors to **check for overfitting** by also visualziing the closest training set example to each generated sample - this could be in the rgb space as well as some perceptual space - look at the sketch retrieval literature or just VGG features.\n\nAnother thing that will help both analyze the dataset quality and the generation quality is to have a big 10x10 grid from both sets (trainig and generation) somehwere in the paper\n\n**Missing citations**\n\nThe missing citations are \n1. @Inproceedings{Paschalidou2021NEURIPS,\n  author = {Despoina Paschalidou and Amlan Kar and Maria Shugrina and Karsten Kreis and Andreas Geiger and Sanja Fidler},\n  title = {ATISS: Autoregressive Transformers for Indoor Scene Synthesis},\n  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},\n  year = {2021}\n}\n2. @inproceedings{10.1145/3588432.3591561,\nauthor = {Para, Wamiq Reyaz and Guerrero, Paul and Mitra, Niloy and Wonka, Peter},\ntitle = {COFS: COntrollable Furniture Layout Synthesis},\nyear = {2023},\nseries = {SIGGRAPH '23}\n}", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "E0rT9kFeXV", "reviewer": "Reviewer_FQwM", "summary": "This paper proposes a method using Diffusion Models and  Set Transformer for generating sketchs of Anime girls in vector format.\nThe training data is obtained by transfering raster images into vector curves.\nThe generation module contain two parts, the Stroke cloud Representation Module (SRM) and the Latent Stroke cloud Generator (LSG). \nThe LSG module is a diffusion model, which serves to sample latent codes. Each latent code corresponds to a sketch result. \nThe SRM module contain two parts:\nThe first part is a Set Transformer, which serves as an encorder to encode training datas into latent codes.\nThe second part is a diffusion model, which serves as an decorder to generate sketch strokes conditioned on the latent code.", "questions": "This question is just for discussion (doesn't affect my rating):\nHow do you critique the technique route that transfers raster images into vectors? Just as your method of establishing your dataset.\nI mean, people can use raster image generation tools such as Stable Diffusion to generate a raster image, and then transfer it into SVG format (the recent work of VectorFusion follows this idea). Compared with this technique route, what's the advantage of your technique route? Or in other words, do you think directly generating vectors is more prospoective?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The idea of applying the Set Transformer to encode the stroke data is the most significant advantage of this paper. The nature of the Set Transformer theoretically guarantees permutation-invariant, so that all the strokes in one sketch can be encoded into a latent code to represent the sketch without worrying about the order of the strokes. This encoder also works even if the stroke number varies among the sketches in the dataset.\n\n2. Also thanks to the Set Transformer, the decoder of the SRM module can sample arbitrary numbers of strokes to constitute a sketch.\n\n3.  Although this paper focuses on the topic of sketch generation. I see the potential of this method to be applied to other artistic styles such as oil painting, just needs to change the stroke model and the dataset.", "weaknesses": "1. I think the main weakness of this paper is the poor visual quality, which is far from artistic application. And the stroke design of the Bezier curve is too simple.  Could the Bezier curve contain more parameters such as width and transparency?\n\n2. Lack of comparison. I do recognize that there may be no methods similar to your technique route. But you should compare with at least one method related to the topic of sketch generation, even though there may be raster-based such as GANs. \n\n3. There seems to be a flaw in the experiment of section 4.4, figure 8. I can see the sketches corresponding to the interpolated results in Figure 8 (the middle ones) are obviously not in the correct domain. This may be because your interpolation function does not fit this distribution (and I didn't see your interpolation function), which leads to the interpolated latent codes are not on their true distribution（you can analogy the manifold of Swiss Roll).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bbKys3Jw8b", "reviewer": "Reviewer_LS4E", "summary": "The idea is to model a vector drawing as a _set_ of strokes.\nDrawings are created by a diffusion process to create a “stroke cloud”.\nThe strokes are represented in a low-dimensional latent space, making them conditionally independent.\nThe paper tackles an interesting problem in a novel and principled way.\nIt produces reasonable results.\nIt is of publishable quality.", "questions": "How well do you you expect the system to generalise to?\n* other objects\n* other points of view\n* other styles\n* the true complexity of real strokes\n\nWhat citable evidence do you have that vectors relate to creativity and pixels to perception?\n\nWhy did you conduct no experiment?\n\nWhy have you not cited any paper prior to 2015, when there is plenty of relevant work?\n\nWhat is your defence fot the claim that \"the primary limitation lies in the probabilistic nature of the reconstruction process\"?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Introduction:\nDifferentiates between vector and pixel representations, and claims the former is pressed (somehow)\non creativity while the latter is based on perception. These claims I have weak sympathy with, but\nwhich I don’t fully agree with (surely people draw with lines/vector because that is how they perceive\nthe world). The are made without any citation into either the perceptual or artistic / art-history literature.\nAnd they pervade the first part of the paper, making slightly uncomfortable reading for me - feeling that\nthe argument made may even mislead. I’d like to see it changed, but given my greater sympathy towards\nthe use of vectors over pixels, it’s not something I will insist upon other than “think again about the argument”.\n\nRelated Works:\nThe paper touches upon only a very small fraction of the available literature, and fails to cite any paper prior to 2015 - even though there is a large body of work (see the non-photorealistic rendering literature).\nMore on this later.\n\nMethod\nThe method is to represent a drawing with a set of strokes, the exact number does not matter.\nI very much like the fact the authors acknowledge strokes are not independent, but use \nDe-Finetti’s Theorem of Exchangeability - which I am not familiar with (so thanks for the introduction)\nso that they are conditionally independent.\n\nI am less convinced by the use of quadratic curves to represent strokes. Strokes can be long\nand very complex. Example - a drawing of curly hair may use long, looped strokes. And strokes vary \nin width, media density and much more. I understand this is a first step - but some text that acknowledges\nthe very severe limitations the method operates under is, i think necessary so that the contribution can\nbe more fairly understood.\n\nIt’s not clear to me why the generator produces a face (in the same view angle, or its mirror, and\nwith the same crop window). This again is very limiting.\n\nResults:\nThe authors make no experiments at all. This is a significant weakness in the paper.\nRather they show some images that have been generated, and then claim the images look good.\nUnfortunately for the authors, others (like me) have a different position.\nI am not convinced the drawings are “high quality” as claimed.\nEspecially when compared to the NPR work my view is that output produced in recent years is low quality.\nAnd no work I have seen in comparable in quality to human art.\n\nThe fact drawing are usually made of some actual thing, like a face or a dog.\nThe paper never mentions how to control the output to direct it to a particular noun class.\n(Does the noun class impact the embedding into latent space? I guess so - it depends on S)\nAll the images are of faces, and all in “manga” style.\nThis means the paper is not clear on its generality.\nDo you have examples of other noun classes?\n\nConclusion:\nI found the conclusion rather limited. I certainly disagree with the claim that \n  “the primary limitation lies in the probabilistic nature of the reconstruction process”.\nIn fact the primary limitation is much more likely to be that the system makes no use of\nsemantic information, other than possibly implicitly via training. It is this issue, rather than any\nother, that has constrained progress in this area.\n\nSummary:\nI enjoyed the paper, in part because of the controversial claims it makes, but mostly because it\ntakes on a difficult problem in an imaginative way. That said, I feel the authors would do themselves and\nthe field much great justice if they were to resist their claims. I strongly recommend looking at some\nreal drawings, ideally “in the flesh” - and not just manga images of 3/4 faces, heavily cropped, drawn\nwith short quadratic curves of uniform width and density, on a perfectly flat surface.\n\nThe paper lacks any experiment, which was once common but is far less so now. One obvious\nexperiment is to conduct some kind of Turing test, of which there are many variants in the NPR/NST\nliterature (especially the more recent). As it stands the paper lack rigour - a rigour which I suspect would\nlead the authors to question some of their more controversial (for me) claims.\n\nThe paper makes a contribution and is publishable.", "weaknesses": "Please see above.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)