ChiroDiff: Modelling chirographic data with Diffusion Models
Authors: Ayan Das
Yongxin Yang
Timothy Hospedales
Tao Xiang
Yi-Zhe Song
Venue: ICLR 2023 poster (2023)
Keywords: chirographic data
continuous-time
diffusion model
generative model

Abstract: Generative modelling over continuous-time geometric constructs, a.k.a $chirographic\ data$ such as handwriting, sketches, drawings etc., have been accomplished through autoregressive distributions. Such strictly-ordered discrete factorization however falls short of capturing key properties of chirographic data -- it fails to build holistic understanding of the temporal concept due to one-way visibility (causality). Consequently, temporal data has been modelled as discrete token sequences of fixed sampling rate instead of capturing the true underlying concept. In this paper, we introduce a powerful model-class namely Denoising\ Diffusion\ Probabilistic\ Models or DDPMs for chirographic data that specifically addresses these flaws. Our model named "ChiroDiff", being non-autoregressive, learns to capture holistic concepts and therefore remains resilient to higher temporal sampling rate up to a good extent. Moreover, we show that many important downstream utilities (e.g. conditional sampling, creative mixing) can be flexibly implemented using ChiroDiff. We further show some unique use-cases like stochastic vectorization, de-noising/healing, abstraction are also possible with this model-class. We perform quantitative and qualitative evaluation of our framework on relevant datasets and found it to be better or on par with competing approaches.

Published as a conference paper at ICLR 2023
CHIRODIFF: MODELLING CHIROGRAPHIC DATA WITH DIFFUSION MODELS
Ayan Das!?, Yongxin Yang Timothy Hospedales Tao Xiang! & Yi-Zhe Song!
SSketchX, CVSSP, University of Surrey; FiFlyTek-Surrry Joint Research Centre AI; QQeenn Mary University of London; *University of Edinburgh, 'Samsung Al Centre, Cambridge
a. @ surrey.ac.uk, yongxin. qmullaac.uk, ciapeddles@eedaa..o (t.iiang, y.song) @ surreey.ac.uk
ABSTRACT
Generative modelling over continuous-time geometric constructs, a.k.a chiro- graphic data such as handwriting, sketches, drawings etc., have been accom- plished through autoregressive distributions. Such strictly-ordered discrete fac- torization however falls short of capturing key properties of chirographic data it fails to build holistic understanding of the temporal concept due to one-way visi- bility (causality). Consequently, temporal data has been modelled as discrete to- ken sequences of fixed sampling rate instead of capturing the true underlying con- cept. In this paper, we introduce powerful model-class namely Denoising Dif- fusion Probabilistic Models or DDPMs for chirographic data that specifically ad- dresses these flaws. Our model named "CHIRDDIF", being non-autoregressive, learns to capture holistic concepts and therefore remains resilient to higher tem- poral sampling rate up to a good extent. Moreover, we show that many important downstream utilities (e.g. conditional sampling, creative mixing) can be flexi- bly implemented using CIFRODIFF. We further show some unique use-cases like stochastic vectorization, de dee-niiiing/healing. abstraction are also possible with this model-class. We perform quantitative and qualitative evaluation of our framework on relevant datasets and found it to be better or on par with competing approaches.
INTRODUCTION
Chirographic data like handwriting, sketches, drawings etc. are ubiquitous in modern day digi- tal contents, thanks to the widespread adoption touch screen and other interactive devices (e.g. AR/VR sets). While supervised downstream tasks such data like sketch- based image retrieval (SBIR) (Liu et al., 2020; Pang et al., 2019), semantic segmentation (Yang et al., 2021; Wang et al., 2020), classification (Yu et al., 2015; 2017) continue flourish due to higher commercial demand, unsupervised generative modelling remains slightly under-explored. Recently however, with the advent of large-scale datasets, generative modelling of chirographic data started gain traction. Specifically, models have been trained on generic doodles/drawings data (Ha & Eck, 2018), or more "specialized" entities like fonts (Lopes al., 2019), diagrams (Gervais al., 2020; Aksan et al., 2020). SVG Icons (Carlier et al., 2020) etc. Building unconditional neural generative models not only allows understanding the distribution of chirographic data but also enables further downstream tasks (e.g. segmentation, translation) by means of conditioning.
8
5
5
1
3
2
5
4
OF
Figure Draw!.
Unconditional samples from CHIRODIFF trained on VMNIST, KanjiVG and Quick,

---

Published as a conference paper at ICLR 2023
Latent space interpolation (Top) with CHIRODIFF using DDIM sampler and (Bottom) with auto-regressive model. CHIRODIFF'S latent space is much more effective with compositional structures for complex data.
Figure 2:
By far; learning neural models over continuous-time chirographic structures have been facilitated broadly by two different representations grid-based raster image and vector graphics. Raster for- mat, the de-facto representation for natural images, has served as an obvious choice for chirographic structures (Yu et al., 2015; 2017). The static nature of the representation however does not pro- vide the means for modelling the underlying creative process that is inherent in drawing. "Creative models", powered by topology specific vector formats (Carlier et al., 2020; Aksan al., 2020; Ha & Eck, 2018; Lopes et al., 2019; Das et al., 2022), on the other hand, are specifically motivated mimic this dynamic creation process. They build distributions of a chirographic entity (e.g.. sketch) X with a specific topology (drawing direction, stroke order etc), i.e. (x) Majority of the creative models are designed with autoregressive distributions (Ha Eck, 2018; Aksan et al., 2020: Ribeiro et al., 2020). Such design choice is primarily due to vector formats having variable lengths, which is elegantly handled by autoregression. Doing so, however, restrict the model from gaining full visibility of the data and fails to build holistic understanding of the temporal concepts. A simple demonstration of its latent-space interpolation confirms this hypothesis (Figure 2). The other possibility is to drop the ordering/sequentiality of the points entirely and treat chirographic data as 2D point-sets and use prominent techniques from 3D point-cloud modelling (Luo & Hu, 2021;;;; Cai et al., 2020). However, point-set representation does not fit chirographic data well due to its inherently unstructured nature. In this paper. with CHIRODIFF, we find a sweet spot and propose a framework that uses non-autoregressive density while retaining its sequential nature.
Another factor in traditional neural chirographic models that limit the representation is effective handling of temporal resolution. Chirographic structures inherently continuous-tima entities as rightly noted by Das et al. (2022). Prior works like SketchRNN (Ha & Eck, 2018) modelled continuous-time chirographic data discrete token sequence or motor program. Due to limited vis- ibility. these models do not have means to accommodate different sampling rates and are therefore specialized to learn for one specific temporal resolution (seen during training), leading to the loss of spatial/temporal scalability essential for digital contents. Even though there have been attempts (Aksan et al., 2020; Das et al., 2020) to directly represent continuous-time entities with their un- derlying geometric parameters, most of them still possess some form of autoregression. Recently, SketchODE (Das et al., 2022) approached to solve this problem by using Neural ODE (abbreviated as NODE) (Chen et al., 2018) for representing time-derivative of continuous-time functions. How- ever, the computationally restrictive nature of NODE's training algorithm makes it extremely hard to train and adopt beyond simple temporal structures. CHIRODIFF, having visibility of the entire sequence, is capable of implicitly modelling the sampling rate from data and consequently is robust to learning the continuous-tin temporal concept that underlies the discrete motor program. In that regard, CHIRODIFF outperforms Das et al. (2022) significantly by adopting a model-class superior in terms of computational costs and representational power while training on similar data.
We chose Denoising Diffusion Probabilstic Models (abbr. as DDPMs) as the model class due to their spectacular ability to capture both diversity and fidelity (Ramesh et al., 2021; Nichol et al., 2022). Furthermore, Diffusion Models are gaining sigmificant popularity and nearly replacing GANs in wide range of visual synthesis tasks due to their stable training dynamics and generation quality. A surprising majority of existing works on Diffusion Model is solely based or specialized to grid- based raster images, leaving important modalities like sequences behind. Even though there are some isolated works on modelling sequential data, but they have mostly been treated as fixed-length entities (Tashiro et al., 2021). Our proposed model, in that regard, is one of the first models to exhibit the potential to apply Diffusion Model on continuous -itie entities. To this end, our generative model generates X by transforming a discretized brownian motion with unit step size.
We consider learning stochastic generative model for continuous-ting chirographic data both in un- conditional (samples shown in Figure 1) and conditional manner. Unlike autoregressive models, CHIRODIFF offers way draw conditional samples from the model without an explicit encoder
2

---

Published as a conference paper at ICLR 2023
when conditioned on homogeneous data (see section 5.4.2). Yet another similar but important ap- plication we consider is stochastic vectorization, i.c. sampling probable topological reconstruction X given a perceptive input R(X) where R is converter from vector representation to perceptive representation (e.g. raster image or point-cloud). We also learn deterministic mapping from noise to data with variant of DDPM namely Denoising Diffission Implicit Model or DDIM which al- lows latent space interpolations like Ha & Eck (2018) and Das et al. (2022). A peculiar property of CHIRODIFF allows a variant of the traditional interpolation which we term as "Creative Mixing", which do not require the model to be trained only on one end-points of the interpolation. We also show a number of unique use-cases like denoising/healing (Su al., 2020; Luo & Hu, 2021a) and controlled abstraction (Muhammad et al., 2019; Das et al., 2022) in the context of chirographic data. As a drawback however, we loose some of the abilities of autoregressive models like stochastic completion etc.
In summary, we propose a Diffusion Model based framework, CHIRODIFF, specifically suited for modelling continuous-time chirographic data (section 4) which, so far, has predominantly been treated with autoregressive densities. Being non-autoregreesive, CHRRODIFF is capable of capturing holistic temporal concepts, leading to better reconstruction and generation metrics (section 5.3). To this end, we propose the first diffusion model capable of handling temporally continuous data modal- ity with variable length. We show plethora of interesting and important downstream applications for chirographic data supported by CHIRODIFF (section 5.4).
2 RELATED WORK
Causal auto-regressive recurrent networks (Hochreiter & Schmidhuber, 1997: Cho et al., 2014) were considered to be natural choice for sequential data modalities due to their inherent ability to encode ordering. It was the dominant tool for modelling natural language (NLP) (Bowman et al., 2016), video (Srivastava et al., 2015) and audio (van den Oord et al., 2016). Recently, due to the break- throughs in NLP (Vaswani et al., 2017), interests have shifted towards non-autoregressive models even for other modalities (Girdhar et al., 2019; Huang et al., 2019). Continuous-time chirographic models also experienced similar shift in model class from LSTMs (Graves, 2013) to Transformers (Ribeiro et al., 2020; Aksan et al., 2020) in terms of representation learning. Most of them however, still contains autoregressive generative components (e.g. causal transformers). Lately, set struc- tures have also been experimented with (Carlier et al., 2020) for representing chirographic data as a collection of strokes. Due to difficulty with generating sets (Zaheer et al., 2017), their objective function requires explicit accounting for mis-alignments. CHIRODIFF finds a middle ground with the generative component being non-autoregressive while retaining the notion of order. A recent un- published work (Luhman & Luhman, 2020) applied diffusion models out-of-the-bos to handwriting generation, although it lacks right design choices, explanations and extensive experimentation.
Diffusion Models (DM), although existed for a while Sohl--Dicsstein et al., 2015), made a break- through recently in generative modelling (Ho et al., 2020; Dhariwal & Nichol, 2021; Ramesh et al., 2021; Nichol et al., 2022). They are arguably by now the de-facto model for broad class of im- age generation (Dhariwal & Nichol, 2021) due to their ability to achieve both fidelity and diversity. With consistent improvements like efficient samplers (Song et al., 2021a; Liu et al., 2022), latent- space diffusion (Rombach al., 2021), classifier(-free guidance (Ho & Salimans, 2022; Dhariwal & Nichol, 2021) these models are gaining traction in diverse set of vision-language (VL) problem. Even though DMs are generic in terms of theoretical formulation, very little focus have been given so far in non -on-image modalities (Lam et al., 2022; Hoogeboom et al., 2022; Xu et al., 2022).
3 DENOISING DIFFUSION PROBABILISTIC MODELS (DDPM)
DDPMs (Ho et al., 2020; Sohl-Dickstein et al., 2015) are parametric densities realized by stochas- tic "reverse diffusion" process that transforms a predefined isotropic gaussian prior p( (()r) ((Tr; 0, I) into model distribution pe (Xo) by de-noising it in T discrete steps. The sequence of T parametric de-noising distributions admit markov property, i.c (e(x-11XXer) = pe (X,,iIXX)) and can be chosen as gaussians
et al., 2015) as long as T is large enough. With the model parameters defined as 6, the de- de-noising conditionals have the form
Po Po(X_11Xx)) (X Mo 1). (x(X
3

---

Published as a conference paper at ICLR 2023
Brownian Motion
De (V
Resl data
P
7
Sampled
t=0
Forward Diffusion
T
T
Reverse Diffusion
Figure 3: The forward and reverse diffusion chirographic data. The "disconnected lines" effect is due to the pen-bits being diffused together. We show the topology by color map (black to yellow).
Sampling can be performed with DDPM sampler (Ho et al., 2020) by starting at the prior XT p(Xr) and rumning ancestral sampling till t using pa- (Xi) where 8* denotes a set of trained model parameters.
Due to the presence of latent variables X1.r. it is difficult to directly optimize the log-likelihood of the model (o().). Practical training is done by first simulating the latent variables by given "forward diffusion" process which allows sampling X1r by means
(X,|)x)) = N (X (1 -x+)T)
with O4 E (0, 1), a monotonically decreasing function of i, that completely specifies the for- ward noising process. By virtue of Eq. 2 and simplifying the reverse conditionals in Eq. 1 with Eg t), == IfI, Sohl-Dickstein et al. (2015); Ho et al. (2020) derived an approximate variational bound Caimple (0) that works well in practice
Caimple (0) = E Exo Xx) c-V().)) | Ea (X e),
where reparameterized Eq. 2 is used to compute a "noisy" version of Xo Xx(Xo, e) = Xo + /I OyE. Also note that the original parameterization ( ,, t) is modified in favour of Ee(XX, t). an estimator to predict the noise given noisy sample X, at any step Please note fr 60 (X,,,)) where Br o1
that they are related as no Mo(X,,)) vI- Xx
Or
DIFFUSION MODEL FOR CHIROGRAPHIC DATA
where the j-th point is (i) E IR? and pاح) E {-1, 1} is a binary bit denoting the pen state, signaling an end of stroke. This representation is popularized by Ha & Eck (2018) and known as Three-point format. We employ the same pre- pre-processing steps (equispaced resampling, spatial scaling etc) laid down by Ha & Eck (2018). Note that the cardinality of the sequence [X] may vary for different samples.
Just like traditional approaches, use the polyline sequence X X= [... (x(0), p
CHIRODIFF is fairly similar to the standard DDPM described in section 3, with the sequence X treated vector arranged by particular topology. However, we found it beneficial not di- rectly use absolute points sequence X but instead use velocities V [.... (y(i), pú1) ,..- ], where v0i x(++1) x which can be readily computed using crude forward/backward differ- ences. Upon generation, we can restore its original form by computing x(i) Ey v('), By modelling higher-order derivatives (velocity instead of position), the model focuses on high-level concepts rather than local temporal details (Ha & Eck, 2018; Das et al., 2022). We may use X and V interchangeably as they can be cheaply converted back and forth at any time.
Please note that we will use the subscript i denote the diffusion step and the superscript (i) to denote elements in the sequence. Following section 3, we define CHIRODIFT, our primary chi- rographic generative model (V) also as DDPM. We use forward diffusion process termed as "sequence-diffusion" that diffuses each element (v p()) independently analogous to Eq. 2
(V/VV) -I () - ; D) q(p."
1) II 1 ( -oo vorpo .1 a.))
g() ()))), with
4

---

Published as a conference paper at ICLR 2023
Forwrr Dihusion (Lmer samping)
Revers Oitusion (Higher sempling)
Figure The reverse process started with higher cardinality |X] than the forward process.
Consequently, the prior at = T has the form ( 11 a(v ( I1 (PT (3) where individual
elements are standard normal, i.e. q(v])) -((p) = I., Note that we treat the binary pen state just like a continuous variable, an approach recently termed as "analog bits" by Chen et al. (2022). Our experimentation show that it works quite well in practice. While generating, we map the analog bit to its original discrete states {-1, by simple thresholding at p=0.
With the reverse sequence diffuuion process modelled as parametric conditional gaussian kernels similar to section 3, i.c. PV(Vi-IVV) == (V M4 Vi, t), I) and analogous change in parame- terization (from μig to EB), we can minimize the following loss
Csimple (0) Evorn(Vo), t-M([, e~V().)) ||ce cV(V(((..c,,,))1
(3)
With a trained Eg., we can run DDPM sampler as 12-1 pa- (V,-1 V/) (V)) (refer to section 3) iteratively for 1. deterministic variant, namely DDIM (Song et al., 2021a), can also be used as
V1 vI arEs (Vz,t) +
Ví-1 Va--1
(Ve,t)
(4)
Vac
Unlike the usual choice of U-Net in pixel-based perception models (Ho al., 2020; Song et al., 2021b; Ramesh et al., 2021; Nichol et al., 2022). CHIRODIFF requires a sequence encoder as (v,V, () in order preserve and utilize the ordering of elements. We chose encode each element in the sequence with the entire sequence as context, i.e. co(v("), Vi,t). Two prominent choices for such functional form are Bi-directional RNN (Bi-RNN) and Transformer encoder (Lee et al., 2019) with positional embedding. We noticed that Bi-RNN works quite well and provides much faster and better convergence.
design choice we found beneficial is to concatenate the absolute positions X₁ along with V/ to the model, i.e. Eg(-, V;: tt), exposing the model to the absolute state of the noisy data at t instead of drawing dynamics only. Since X, can be computed from V, itself, we drop X, from the function arguments now onward just for notation brevity. Please note that the generation process non-causal as it has access to the entire sequence while diffusing. This gives rise to non-autoregressive model and thereby focusing on holistic concepts instead of low-level motor program. This design allows the reverse diffusson (generation) process to correct any part of sequence from earlier mistakes, which is not possible in auto-regressive models.
Transforming "Brownian motion" into "Guided motion": CHIRODIFF'S generation pro- cess has an interesting interpretation. Recall that the reverse diffusion process begins at Vr = [..., () (2) ........ where each velocity element V (j) *N(O I). Due to velocity-position .pr
encoding described above, the original chirographic structure is then xf = E, vl which, by definition, is a discretized brownian motion with unit step size. With the reverse process unrolled in time, the brownian motion with full randomness transforms into motion with structure. leading to realistic data samples. We illustrate the entire process in Figure 3.
Length conditioned re-sampling: A noticeable property of CHIRODIF'S generative process is that there is no hard conditioning on the cardinality of the sequence |X| or [V] due to our choice of the parametric model ( tt. As result, we can kick-off the generation (reverse diffusiion) process by sampling from a prior p(Vr) what the model was trained on. We hypothesize and empirically show (in section 5.3) that if trained to optimiality, the model indeed captures high level geometric concepts and can generate similar data with higher sampling rate (refer to Figure 4) with relatively less error. We credit this behaviour to the accessibility of the entire sequence Vi (and additionally X,) to the model co(-). With the full sequence visible, the model can potentially build an internal (implicit) global representation which explains the resilience increased temporal sampling resolution.
(iipg(v))) )g(x)") of any length L, potentially higher than
5

---

Published as a conference paper at ICLR 2023
SJNN
SAN CosE
Kanffi
5,000
5-2CE
40-
QD
CoRE
6.3
0.3
20
0.2
02
0.01
Figure 5: (A, B, C) Reconstruction CD against sampling rate factor. (D) Relative convergence time & sampling time (transparent bars) w.r.t our method. (E) FID of unconditional generation (averaged over multiple classes for QD).
5
EXPERIMENTS & RESULTS
5.1 DATASETS
VectorMNIST VMNIST (Das et al., 2022) is vector analog of traditional MNIST digits dataset. It contains 10K samples of 10 digits ("0' to '9') represented in polyline sequence format. We use 80-10-10 splits for our all our experimentation.
KanjiVG vector dataset containing Kanji characters. We use a preprocessed version of the dataset which converted the original SVGs into polyline sequences. This dataset is used in order to evaluate our method's s effective on complex chirographic structures with higher number of strokes.
Quick, Draw! (Ha & Eck, 2018) is the largest collection of free-hand doodling dataset with casual depictions of given concepts. This dataset is an ideal choice for evaluating method's '' effee- tiveness on real noisy data since it was collected by means of large scale crowd-sourcing. In this paper, we use the following categories: {cat, crab, bus, mosquito, fish, yoga, flower).
5.2 IMPLEMENTATION DETAILS
CHIRODIFF'S forward process, just like traditional DDPMs, uses a linear noising schedule of {Smin = 10-4. 1000/ /T, ,T 2 x 10-2. 1000/T] as found out by Nichol & Dhariwal (2021); Dhariwal & Nichol (2021) to be quite robust. We noticed that there isn' 't much performance differ- ence with different diffusion lengths, so we choose a standard value of T = 1000. The parametric noise estimator co(v!") Vi, t) is chosen to be bi-directional GRU encoder (Cho et al., 2014) where each element of the sequence is encoded while having contextual information from both direction of the sequence, making it non-causal. We use a 2-layer GRU with D 48 hidden units for VM- NIST and 3-layer GRU for QuickDraw (D = 128) and KanjiVG (D = 96). We also experimented with transformers with positional encoding but failed to achieve reasonable results, concluding that positional encoding is not a good choice for representing continuous time. We trained all of our models by minimizing Fg.. 3 using AdamW optimizer (Loshchilov & Hutter, 2019) and step-wise at every epoch e where " = 6 10-3 The diffusion time-step
LR scheduling of Yo 0.9997- made available to the model by concatenating sinusoidal positional embeddings
t e 1,2, (Vaswani et al., 2017) into each element of a sequence at every layer. We noticed the importance of reverse process variance of in terms of generation quality of our models. We found of = 0.8,3, to work well in majority of the cases, where 3. 1-0t-1 Br as defined by Ho et al. (2020) to be true variance of the forward process posterior. Please refer to the project page" for full source code.
5.3 QUANTITATIVE EVALUATIONS
In order to assess the effectiveness of our model for chirographic data, perform quantitative eval- uations and compare with relevant approaches. We measure performance in terms of representation learning, generative modelling and computational efficiency. By choosing proper dimensions for competing methods//rcciiecttrrss. we ensured approximately same model capacity (# of parame- ters) for fare comparison.
Original VanjiVG: kanjivg, haagiiig.tagaii..ne
PPee- Preepprcessed KanjiVG:
*Our project page: http://ayaanaas.m/cchicceo

---

Published as a conference paper at ICLR 2023
Reconstruction We construct condi-
oweer Swazize DOE Auttvebre Darsann
tional variant of CHIRODIFF with an en-
coder yy being a traditional Bi-GRU for
fully encoding a given data sample V into latent code z. The decoder, in our case, is a diffusion model described in sec-
tion 4. We sample from the conditional
y
model Po z = (V)) which is effec- tively same as standard DDPM but with the noise-estimator 6g- (V,,,, addition- ally conditioned on z. We expose the la- tent variable z to the noise-estimator by simply concatenating it with every ele- ment j at all timestep t E [1, We also evaluate CHIRODUF'S ability to adopt to higher tem- poral resolution while sampling, proving our hypothesis that it captures concepts at a holistic level. We encode sample with εv, and decode explicitly with higher temporal sampling rate (refer to section 4). We compare our method with relevant frameworks like SketchODE (Das et al., 2022), SketchRNN (Ha & Eck, 2018) and CoSE (Aksan et al., 2020). Since autoregressive models like SketchRNN has no explicit way to increase temporal resolution, we train different models with re- sampled data, which is already disadvantageous. We quantitatively compare them with Chamfer Distance (CD) (Qi et al.. 2017) (ignore the pen-up bit) for conditional reconstruction. Figure 5 shows the reconstruction CD against sampling rate factor (multiple of the original data cardinality) which shows the resilience of our model against higher sampling rate. SketchRNN being autore- gressive, fails at high sampling rate (longer sequences), as visible in Figure 5(A, & C). CoSE and SketchODE being naturally continuous, has a relatively flat curve. Also, we couldn't 't reasonably train SketchODE on the complex VanjiVG dataset (due to computational and convergance issues) and hence omitted from Figure 5. Qualitative examples of reconstruction shown in Fig. 6.
Figure 6: 1st and 2nd column for each example depicts sampling rate 1 & 2 respectively while reconstructing.
Generation We assess the generative performance of CHIRODIFF by sampling unconditionally (in 50 steps with DDIM sampler) and compute the FID score against the real data samples. Since the original inception network is not trained on chirographic data, we train our own on Quick, Draw! dataset (Ha & Eck, 2018) following the setup of (Ge et al., 2021). We compare our method with SketchRNN (Ha & Eck, 2018) and CoSE (Aksan et al., 2020) on all three datasets. Quantitative results in Figure 5(E) show a consistent superiority of our model in terms of generation FID. Quick- Draw FID values are averaged over the individual categories used. Qualitative samples are shown in Figure 1 and more in appendix A. with potential drawbacks highlighted. We also conducted small scale user study to validate our generated samples and found our method to be superior (see appendix A.3 for details).
Computational Efficiency We also compare method with competing methods in terms of ease of training and convergence. We found that our method, being from Diffusion Model family, enjoys easy training dynamics and relatively faster convergence (refer Figure 5(D)). We also provide approximate sampling time for unconditional generation.
5.4
DOWNSTREAM APPLICATIONS
5.4.1
STOCHASTIC VECTORIZATION
An interesting use-case of generative chirographic model is stochastic vectorization, i.e. recreating plausible chirographic structure (with topology) from given perceptive input. This application is intriguing due to human's 's innate ability to do the same with ease. The recent success of diffusion models in capturing distributions with high number of modes (Ramesh et al., 2021; Nichol et al., 2022) prompted us to use it for stochastic vectorization, a problem of inferring potentially multi- modal distribution. We simply condition our generative model on perceptive input X' R(X)= (x())(()) p(0) E X). i.e. we convert the sequence into point-set (also densely resample them as part of pre-processing). We employ a -et-transformer encoder E Ex(-) with max max-pooling aggregator (Lee et al., 2019) to obtain latent vector z and condition the generative model similar to section 5.3, Le. (V = {x)). We evaluated the conditional generation with Chamfer Distance (CD) on test set and compare with Das et al. (2021) (refer to Figure 7).
7

---

Published as a conference paper at ICLR 2023
5.4.2 IMPLICIT CONDITIONING
Unlike explicit conditioning mechanism (includes an encoder) described in section 5.3, CHIRODIFF allows a form of "Im- plicit Conditioning" which requires no explicit encoder. Such conditioning is more stochastic and may not be used for re- construction, but can be used to sample similar data from a pre-trained model po-. Given a condition Xgoad (or Vgoand, we sample a noisy version at = To < ' (a hyperparam- eter) using the forward diffusion (Eq. 2) process as Vi.on var, V/oond + vI or, c where the trained model pg- to gradually de-noise Vi.and We run the reverse process from = Te till =
5.30
WARIS
Karjus
0.00
Ours (tech sec
Douticuna
5
1
(O.II). We then utilize
2o
Figure 7: Stochastic vectoriza- tion. Note the different topolo- gies (color map) of the samples.
Vi-1 pa- (V), for T. t> O, with Vr.
Vronn
The hyperparameter T. controls how much the generated samples correlate the given condition. By starting the reverse process at & = Te with the noisy condition, we essentially restrict the generation be within region of the data space that resembles the condition. Higher the value of T., the generated samples will resemble the condition more (refer to Figure 8). We also classified the generated samples for VMNIST & Quick, Draw! and found it to belong the same class as the condition 93% of the time in average.
5.4.3 HEALING
The task of healing is more prominent in 3D point-cloud literature (Luo & Hu, 2021a). Even though the typical chirographic (autoregres- sive) models offer "stochastic completion" (Ha & Eck, 2018), it does not offer an easy way "heal" a sample due to uni-directional gen- eration. It is only very recently, works have emerged that propose tools for healing bad sketches (Su et al., 2020). With diffusion model family, is fairly straightforward to solve this problem with CHIRODIFF. Given a "poor" chi- rographic data Xo. we would like to generate samples from region in pa- (Xo) close to Xo in terms of semantic concept. Surprisingly, this problem can be solved with the "Implicit Condi- tioning" described in section 5.4.2. Instead of real data condition, we provide the poorly drawn data Xo (equivalently Vo) condition. Just as before, with Vr, Vr, in order to sample from healed data distribution around Vo. Th is similar hyperparameter that decides the trade-off between healing the given sample and drifting away from it in terms of high-level concept. Refer to Figure 8 (right) for qualitative samples of healing (with Th = T/5).
5
5
5
5
D
Figure 8: Implicit conditioning and healing.
the reverse process starting t Th
5.4.4 CREATIVE MIXING
Creative Mixing is a chirographic task of merging two high-level concepts into one. This task is usually implemented as latent-space interpolation in traditional autoencoder-style generative models (Ha Eck, 2018; Das et al., 2022). variant of CHIRODIFF that uses DDIM sampler (Song al., 2021a), be used for similar interpolations. We use pre-trained conditional model to decode the interpolated latent vector using V/ as a fixed point. Given two samples Vo, and Vo,, we compute the interpolated latent variable Zinterp (1 8) (Vo,) + (8v (Voz) for any 8 E [0, 1] and run DDIM sampler shown in Eq. 4 with the hoise-estimator Eg- (Ve. t, )interp).
This solution works well for some datasets (KanjiVG & VMNIST; shown in Figure 9 (left)). For others (Quick, Draw), we instead propose a more general method inspired by ILVR (Choi et al., 2021) that allows "mixing" using the DDPM sampler itself. In fact, it allows to perform mixing
8

---

Published as a conference paper at ICLR 2023
g
Figure 9: (Left) Latent-space semantic interpolation with DDIM. (Right) Creative Mixing shown as three-tuples consisting of Xo, Xg and the mixed sample respectively.
without one of the samples (reference sample) being known to the trained model. Given two samples of potentially different concepts, Xo and Xj (or equivalently Vo and Viref,, sample from pre- trained conditional model given Vo. but with a modified reverse process
X X ( + (x;=))
where, V Pa- (V.-1 V1, (Vo), and Vi q( (V7=f (gget
where (-) is a temporal low-pass filter that reduces high-frequency details of the input data along
temporal axis. We implement (.) using temporal 1D convolution of window size w = 7. Please note that for the operation Eq. 5.4.4 to be valid, the sequences must be of same length. We simply resample the conditioning sequence to match the cardinality of the reverse process. The three-tuples presented in Figure (right) shows the mixing between different categories of Quick, Draw!.
5.4.5 CONTROLLED ABSTRACTION
Visual abstraction is relatively new task (Muhammad et al., 2019; Das al., 2022) in chirographic literature that refers to deriving a new distri- bution (possibly with control) that holistically matches the data distribu- tion, but more "abstracted" in terms of details. Our definition to the prob- lem matches that of Das et al. (2022), but with an advantage of being able to use the same model instead of re- training for different controls.
p
G00 g00
6oo
0000
800
b
0
9 +
Figure 10: With reduced reverse process variance of l- generated samples looses high-freyuency details but retains abstract concepts.
The solution of the problem lies in the sampling process CHIRODIFF, which has an abstraction effect when the reverse process variance of is low. We define continuous control k E [0, 1] as hyperparameter and use reverse process variance of of k -B The rationale behind this method is that when k is near zero, the reverse process stops exploring the data distribution and converges near the dominant modes, which are data points conceptually resembling original data but more "canonical" representation (highly likely under data distribution). Figure 10 shows qualitative results of this observation on Quick, Draw!.
CONCLUSIONS, LIMITATIONS & FUTURE WORK
In this paper, we introduced CHIRODIFF, non-autoregressive genera- tive model for chirographic data. ChiroDiff is powered by DDPM and offers better holistic modelling of concepts that benefits many down- stream tasks, some of which not feasible with competing autore- gressive models. One limitation of our approach is that vector represen- tations are more susceptible to noise than raster images (see Figure 11 (Top)). Since we empirically set the reverse (generative) process vari- ance of the noise sometimes overwhelms the model-predicted mean (see Figure 11 (Bottom)). Moreover, due the use of velocities, the ac- cumulated absolute positions also accumulate the noise in proportion its cardinality. One possible solution is to modify the noising process to be adaptive to the generation or the data cardinality itself. We leave this as a potential future improvement on CHIRODIFF.
5 W
Figure II: (Top) Noisy vector and raster data at same (Bottom) Failures due noise.

---

Published as a conference paper at ICLR 2023
REFERENCES
Emre Aksan, Thomas Deselaers, Andrea Tagliasacchi, and Otmar Hilliges. Cose: Compositional stroke embeddings. NeurIPS, 2020.
Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew Dai, Rafal Jozefowicz, and Samy Bengio. Generating sentences from continuous space. CONLL, 2016.
Ruojin Cai, Guandao Yang, Hadar Averbuch-Flor, Zekun Hao, Serge J. Belongie, Noah Snavely, and Bharath Hariharan. Learning gradient fields for shape generation. In ECCV, 2020.
Alexandre Carlier, Martin Danelljan, Alexandre Alahi, and Radu Timofte. Deepsvg: A hierarchical generative network for vector graphics animation. NeurIPS, 2020.
Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. In NeurIPS, 2018.
Ting Chen, Ruixiang Zhang, and Geoffrey E. Hinton. Analog bits: Generating discrete data using diffusion models with self-conditioning. ArXiv, abs/2208. 04200,, 2022.
Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçehre, Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In EMNLP, 2014.
Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon. ILVR: conditioning method for denoising diffusion probabilistic models. In ICCV, 2021.
Ayan Das, Yongxin Yang, Timothy Hospedales, Tao Xiang, and Yi-Zhe Song. generative model for scalable vector sketches. In ECCV, 2020.
Béziersketch: A
Ayan Das, Yongxin Yang. Timothy M. Hospedales, Tao Xiang, and Yi-Zhe Song. Cloud2curve: Generation and vectorization of parametric sketches. In CVPR, 2021.
Ayan Das, Yongxin Yang, Timothy M Hospedales, Tao Xiang, and Yi-Zhe Song. Sketchode: Learn- ing neural sketch representation in continuous time. In ICLR, 2022.
Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat gans on image synthesis. In NeurlPS, 2021.
songwei Ge. Vedanuj Goswami, Larry Zitnick, and Devi Parikh. Creative sketch generation. In ICLR, 2021.
Philippe Gervais, Thomas Deselaers, Emre Aksan, and Otmar Hilliges. The DIDI dataset: Digital ink diagram data. CoRR, 2020.
Rohit Girdhar, João Carreira, Carl Doersch, and Andrew Zisserman. Video action transformer net- work. In CVPR, 2019.
Alex Graves. Generating sequences with recurrent neural networks. 1r08.v1308.000 0850, 2013.
arXiv preprint
David Ha and Douglas Eck. A neural representation of sketch drawings. In ICLR, 2018.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. CoRR, 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS, 2020.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Comput., 1997.
Emiel Hoogeboom, Victor Garcia Satorras, Clément Vignac, and Max Welling. Equivariant diffu- sion for molecule generation 3d. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), ICML., 2022.
10

---

Published as a conference paper at ICLR 2023
Cheng-Zh ZZi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, lan Simon, Curtis Hawthome, Noam Shazeer, Andrew M. Dai, Matthew D. Hoffman, Monica Dinculescu, and Douglas Eck. Music transformer:
Generating music with long-ten term structure. In ICLR, 2019.
Max W. Y. Lam, Jun Wang, Dan Su, and Dong Yu. BDDM: bilateral denoising diffusion models for fast and high-quality speech synthesis. In ICIR, 2022.
Juho Lee. Yoonho Lee, Jungtack Kim, Adam R. Kosiorek, Scungjin Choi, and Yee Whye Teh. Set transformer: A framework for attention-based permutation commutatinninvariiant neural networks. ICML, 2019.
Fang Liu, Changqing Zou, Xiaoming Deng, Ran Zuo, Yu-Kun Lai, Cuixia Ma, Yong-Jin Liu, and Hongan Wang. Scenesketcher Fine-grained image retrieval with scene sketches. In ECCV, 2020.
Luping Liu, Yi Ren, Zhijic Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds. In ICLR, 2022.
Raphael Gontijo Lopes, David Ha, Douglas Eck, and Jonathon Shlens. learned representation for scalable vector graphics. In ICCV, 2019.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
Troy Luhman and Eric Luhman. Diffusion models for handwriting generation. abs/201 1.001110000 100704, 2020.
CoRR,
Shitong Luo and Wei Hu. Score-based point cloud denoising. In ICCV, 2021a.
Shitong Luo and Wei Hu. Diffusion probabilistic models for 3d point cloud generation. In CVPR, 2021b.
Umar Riaz Muhammad, Yongxin Yang, Timothy M Hospedales, Tao Xiang, and Yi-Zhe Song. Goal-driven sequential data abstraction. In ICCV, 2019.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In ICML, 2021.
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. GLIDE: towards photorealistic image generation and editing with text-guided diffusion models. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), JCML, 2022.
K. Pang, Li, Yang, H. Zhang, T. M. Hospedales, Xiang, and Y. -Z. Song. Generalising fine-grained sketch-based image retrieval. In CVPR, 2019.
Charles Ruizhongtai Qi, Hao Su, Kaichun Mo, and Leonidas Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In CVPR, 2017.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In Marina Meila and Tong Zhang (eds.), ICML, 2021.
Leo Sampaio Ferraz Ribeiro, Tu Bui, John P. Collomosse, and Moacir Ponti. Transformer-base representation for sketched structure. In CVPR, 2020.
Sketchformer:
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. CVPR, 2021.
Jascha Dickkstein Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In ICML, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2021a.
11

---

Published as a conference paper at ICLR 2023
Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLR, 2021b.
Nitish Srivastava, Elman Mansimov, and Ruslan Salakhudinov. Unsupervised learning of video representations using Istms. In ICML, 2015.
Guoyao Su, Yonggang Qi, Kaiyue Pang, Jie Yang. and Yi-Zhe Song. Sketchhealer: A graph-to- sequence network for recreating partial human sketches. BMVC, 2020.
Yusuke Tashiro, Jiaming Song, Yang Song, and Stefano Ermon. CSDI: conditional score-based diffusion models for probabilistic time series imputation. In NeurIPS, 2021.
Aäron den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew W. Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. In ISCA, 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017.
Fei Wang, Shujin Lin, Hanhui Li, Hefeng Wu, Tie Cai, Xiaonan Luo, and Ruomei Wang. Multi- column point-cnn for sketch segmentation. Neurocomputing, 2020.
Minkai Xu, Lantao Yu, Yang Song, Chence Shi, Stefano Ermon, and Jian Tang. Geodiff: A geomet- ric diffusion model for molecular conformation generation. In ICLR, 2022.
Lumin Yang. Jiajie Zhuang, Hongbo Fu, Xiangzhi Wei, Kun Zhou, and Youyi Zheng. SketchGNN: Semantic sketch segmentation with graph neural networks. ACM Transactions on Graphics (TOG), 2021.
Qian Yu, Yongxin Yang. Yi-Zhe Song. Tao Xiang, and Timothy Hospedales. Sketch-a-net that beats humans. In BMVC, 2015.
Qian Yu, Yongxin Yang, Feng Liu, Yi-Zhe Song, Tao Xiang, and Timothy Hospedales. Sketch-a-net: A deep neural network that beats humans. IJCV. 122:411-425, 2017.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabás Póczos, Ruslan Salakhutdinov, and Alexander Smola. Deep sets. NeurlPS, 2017.
12

---

Published as a conference paper at ICLR 2023
A APPENDIX
A.1 QUALITATIVE COMPARISONS
We make qualitative comparison of all the competing methods mentioned in this paper. We compare unconditionally generated samples for SketchRNN (Ha & Eck, 2018), CoSE (Aksan et al., 2020) and CHIRODIFF, with reconstructed samples for SketchODE (Das al., 2022) since marily generative model. The qualitative samples given below (figure 12, 13 & 14) for all three datasets from different models provide insights into their capabilities and drawbacks (we explicitly highlighted some of them).
not pri-
SketchRNN
5
65
6
5
748
8
Broken trajectories
4
2
2331
7
5
2 5
779602662
2 7
Less variations
CoSE
662728236 C
ChiroDiff SketchODE
662408565
Too smooth
932926492
5
399739599 349446441
Figure 12: We show qualitative samples from different models trained on VectorMNIST dataset. We notice some common failure case in other models which is mitigated in CHIRODIFF. SketchRNN suffers from broken trajectories, CoSE suffers from less variability due it's stroke-based genera- tion and SketchODE produces overly smooth trajectories due to the truely continuous nature of the underlying representation.
SKetchRNN
7
compositionallty
Broken
*
CoSE
Pepealeed structures
D
ス
ChiroDiff
Figure 13: We show qualitative samples from different models trained on KanjiVG dataset. This dataset is particularly suitable for assessing model's capability to deal with large number of strokes and difficult compositionality. SketchRNN suffers from broken compositionality due lack of holistic representation (causality), CoSE suffers from repeated stroke/structure generation.
13

---

Published as a conference paper at ICLR 2023
Broken end end-of-e fnd--of-ske ske-sk token
Sketch
Less variation
CoSE
Oversmoolhening
StetchODE
ChiraDiff
Figure 14: We show qualitative samples from different models trained on QuickDraw dataset. We noticed similar issues as mentioned for VectorMNIST dataset Fig. 12.
A.2 TABLE FOR QUANTITATIVE RESULTS
In this subsection, we provide the quantitative results of all of our experiments mentioned in the main paper in tabular format (table & in order to complement the graphical results of Fig. 5 & 7.
Datasets
VMNIST
KanjiVG
Quick. Draw!
Scale-factor
1.0
1.5
2.0
1.0
1.5
2.0
1.0
1.5
2.0
SketchRNN (Ha & Eck, 2018)
0.0123
0.0137
0.0190
0.163
0.237
0.413
0.163
0.227
0.373
CoSE (Aksan et al., 2020)
0.02133
0.0235
0.0280
0.143
0.172
0.235
0.133
0.172
0.244
SketchODE (Das et al., 2022)
0.0279
0.0311
0.0346
0.411
0.431
0.474
CHIRODIFF (ours)
0.0121
0.0135
0.0176
0.121
0.155
0.206
0.11
0.136
0.204
Datascts
VMNIST
KanjiVG 2.7
Quick, Draw!
SketchRNN(Ila & Bck, 2018)
1.31
1.3
CoSE (Aksan et al., 2020)
1.42
SketchODE (Das et al., 2022)
6.1
CHIRODIFF (ours)
1.0
1.0
1.0
Datasels
VMNIST
KanjiVG
Quick, Drow!
SketchRNN(HHa Eck, 2018)
0.18
0.26
0.28
CoSE (Aksan al., 2020)
0.31
0.40
0.46
SketchODE (Das et al., 2022)
0.63
0.86
CHIRODIw (ours)
1.0
1.0
1.0
Datasets
VMNIST
KanjiVG
Quici, Draw!
SketchRNN (Ha & Eck, 2018)
13.11
32.12
39.71
CoSE (Aksan et al., 2020)
10.05
23.8
35.46
CHIRODiFF (ours)
10.57
15.31
25.12
Table 1: Three tables above provide the quantitative results complementing fig. 5. The first table shows reconstruction Chamfer Distance (CD) for various methods and datasets. The second and third tables depict the relative convergence and sampling time w.r.t ours. The third table shows unconditional generation FIDs.
14

---

Published as a conference paper at ICLR 2023
Datasets
VMNIST
VGnjiVG
Quick, Draw!
Cloud2Curve (Das et al., 202T)
0.07
0.106
0.230
CHIRODIFF (ours)
0.014
0.172
0.130
Table The above table shows reconstruction CD between methods comparing their stochastic vectorization capabilities. This table complements graphical results of Fig. 7.
A.3 QUALITATIVE USER STUDY
We conduct a small scale user validation study in order to verify the effectiveness of our generative model. We first sampled set of 20 samples from each model, CHIRODIFF, SketchRNN (Ha & Eck, 2018), CoSE (Aksan et al., 2020) and SketchODE as shown in appendix A.1. We show the set of generated samples along with 20 real samples side-by-side them to assess which one is realistic. Below tables shows the # of users identifying the generated samples to be real. As can be inferred from the table, samples generated by CHIRODIFF are labelled to be realistic by more users than competing methods.
users independently and asked
VMNIST
KanjiVG
QuickDraw
SketchRNN (Ha & Eck, 2018)
5
0
5
CoSE (Aksun et al., 2020)
6
3
4
SketchODE (Das et al., 2022)
2
0
CHIRODIFF (ours)
6
5
Table 3: Results of the user study. # of users labelled generated samples to be real.
15


Reviews:
[{"review_id": "xpXpXLYNJWs", "reviewer": "Reviewer_N1HR", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method to model chirographic data using diffusion models. Continuous-time geometric data (i.e., chirographic data) is typically modeled via autoregressive methods which fail to capture holistic information of the temporal data. On the contrary, the proposed ChiroDiff is capable of capturing holistic concepts and remaining resilient to higher temporal sampling rate. To my knowledge, this work is the first to model chirographic data with diffusion models. Both quantitative and qualitative experiments have been conducted to verify the effectiveness of the proposed method in several different applications. ", "summary_of_the_review": "The authors deserve the credit since they are the first to introduce diffusion models to handle the interesting task of chirographic data synthesis. However, there still exist some weaknesses in this paper as mentioned above. Otherwise, I could give it a higher score.", "strength_and_weaknesses": "Strength:\nRecently, the superiority of diffusion models has been proven in the application of image synthesis. However, no work has been done to demonstrate the effectiveness of diffusion models in sequential data synthesis. This paper seems to be the first work to effectively using diffusion models to synthesize chirographic data. Therefore, I think many researchers will be interested in this paper, which makes some valuable academic contributions to the community. Despite some tiny representation problems, the paper is generally well-written and easy to follow. \n\nWeaknesses:\n1) The whole Section 3 provides a detailed description of DDPM, which is already well-known in the literature. I think this section should be shortened, removed, or integrated into Section 4 to emphasize the technical contribution of the proposed ChiroDiff.\n2) Tables with accurate evaluation values should be provided instead of just displaying plots (Fig. 5 and 6). What is worse, no qualitative comparison has been made, and no user study has been conducted. In fact, for the tasks of image/graphic synthesis, qualitative evaluation is often more important than quantitative comparison. Many people want to directly compare the performance of two image/graphic synthesis methods by inspecting their synthesis results. Therefore, I suggest showing more qualitative results obtained by the proposed ChiroDiff and other existing approaches in the manuscript and its supplemental materials. Only in this manner, the advantage and necessity of modeling chirographic data via diffusion models can be clearly verified.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "L-T7NS78luk", "reviewer": "Reviewer_VahX", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes the use of diffusion models (DM), more specifically the Denoising Diffusion Probabilistic Models (DDPMs), for modeling chirographic data such as digital handwriting, sketching, and drawings. In contrast to the mainstream autoregressive approaches on sequential chirographic data, the proposed method, CHIRODIFF, processes the entire sequence in one go and hence better captures holistic concepts over sequences. The paper demonstrates the benefits of the proposed method in a variety of tasks, including de-noising, interpolation, and temporal reconstruction of images or 2D point clouds. CHIRODIFF is also compared with various architectures implementing different model classes, and it achieves better or competitive performance in reconstruction and generation tasks on the VectorMNIST, KanjiVG, and a subset of the QuickDraw datasets.", "summary_of_the_review": "I find this paper useful for the community. It introduces a new model class and compares it against a carefully chosen set of baselines that implement different modeling concepts. Despite the advantages of the underlying diffusion models, there are also some weaknesses, which should be discussed in the rebuttal and incorporated into the paper. I am leaning toward accepting this paper if the authors provide clarifications in the rebuttal and address my concerns. \n\n---------\n**Post rebuttal update:** The rebuttal and the proposed revisions address my concerns. Hence, I am in favor of accepting this paper. ", "strength_and_weaknesses": "Strengths:\n\n1- The paper applies a more powerful and expressive model class, the diffusion models, to chirographic data.\n\n2- The paper is well-motivated around the holistic view achieved by the diffusion model. I think it is a novel approach in terms of how the sequential chirographic data is treated.\n\n3- The proposed model has an interesting interpretation. At T, before any reverse diffusion steps, the chirographic structure is represented as Brownian motion. It basically treats every timestep as an independent data point by ignoring the causality. Nevertheless, it learns to transform this unstructured representation into realistic data samples. This introduces another interesting property that the model can handle sequences with arbitrary lengths and sampling frequencies (at least in theory; see the first weakness related to this). \n\n---\n\nWeaknesses:\n\n1- One of the main claims of this work is the proposed model’s inherent advantage for modeling continuous-time data. I am not sure if this is entirely correct, as the backbone architecture is a Bi-RNN that processes the sequence step-by-step to approximate the noise. How could CHIRODIFF be better than the baselines like SketchODE or the decoder of CoSE? In particular, the latter can reconstruct a sequence at arbitrary resolution. Isn’t the underlying Bi-RNN prone to failure when the sequence length is arbitrarily increased (i.e., out-of-domain problem)?\n\n2- In the stochastic vectorization task, the reconstructions are slightly different from the perceptive input. I am not sure if this is the expected result. While the model could capture different temporal modes and preserve semantics, it changes its appearance. Could it be due to the quality of the latent condition z or the stochasticity of the diffusion model-based decoder? \n\n3- In contrast to the auto-regressive approaches, the proposed model lacks the ability to complete partial drawings. I think this is perfectly fine but it should be discussed in the paper.\n\n4- This is not a weakness per se. Considering that the paper doesn’t introduce major modifications to the original diffusion model formulation, the background section on diffusion models seems a bit redundant. Some space could be saved for additional experiments, especially for ablation on the model’s performance on more complex data such as the hand-drawn diagram dataset of Gervais et al. or on computational efficiency (i.e., sampling-time vs. increasing sequence lengths, design choices, etc.).\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "5QEjr_NxFo", "reviewer": "Reviewer_DeoY", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposed a method for generating online handwriting data based on diffusion models. On several datasets, the proposed method outperforms previous approaches.", "summary_of_the_review": "Although the technical novelty of this work is limited, it is the first work that applies diffusion models to generating temporal data and achieves promising results on several datasets.", "strength_and_weaknesses": "Strength:\n- To my best knowledge, this is the first work that explores diffusion models to generate temporal sequences.\n- The proposed method overcomes the limitations of autoregressive models and achieves better empirical results on several datasets.\n\nWeakness:\n- The method directly adopts a previous diffusion model (DDPM) by replacing UNet with GRU. Thus, the technical novelty of the method is limited.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster