SketchODE: Learning neural sketch representation in continuous time
Authors: Ayan Das
Yongxin Yang
Timothy Hospedales
Tao Xiang
Yi-Zhe Song
Venue: ICLR 2022 Poster (2022)
Keywords: Chirography
Sketch
Free-form
Neural ODE

Abstract: Learning meaningful representations for chirographic drawing data such as sketches, handwriting, and flowcharts is a gateway for understanding and emulating human creative expression. Despite being inherently continuous-time data, existing works have treated these as discrete-time sequences, disregarding their true nature. In this work, we model such data as continuous-time functions and learn compact representations by virtue of Neural Ordinary Differential Equations. To this end, we introduce the first continuous-time Seq2Seq model and demonstrate some remarkable properties that set it apart from traditional discrete-time analogues. We also provide solutions for some practical challenges for such models, including introducing a family of parameterized ODE dynamics & continuous-time data augmentation particularly suitable for the task. Our models are validated on several datasets including VectorMNIST, DiDi and Quick, Draw!.

Published as a conference paper at ICLR 2022
SKETCHODE: LEARNING NEURAL SKETCH REPRESENTATION IN CONTINUOUS TIME
Ayan Das!?, Yongxin Yang" Timothy Hospedales Tao Xiang! & Yi-Zhe Song! "SketchX, CVSSP, University of Surrey, UK
FiFlyTek-Sa TFlyTee--urrry Joint Research Centre on Artificial Intelligence
School of Informatics, University of Edinburgh, UK
dasssuuree..er ac. {yongxin . t t.hospedales @ed ac uk.uk, {t. (t.xiange xiang, y. e.8ong) ac.
ABSTRACT
Learning meaningful representations for chirographic drawing data such as sketches, handwriting, and flowcharts a gateway for understanding and emulat- ing human creative expression. Despite being inherently continuous time data, ex- isting works have treated these as discrete-time sequences, disregarding their true nature. In this work, we model such data as continuous-ting functions and learn compact representations by virtue of Neurul Ordinary Differential Equations. To this end, we introduce the first continuous-time Seq2Seq model and demonstrate some remarkable properties that set apart from traditional discrete-time ana- logues. We also provide solutions for some practical challenges for such models, including introducing a family of parameterized ODE dynamics & continuous- time data augmentation particularly suitable for the task. Our models validated on several datasets including VectorMNIST, DiDi and Quick, Draw
INTRODUCTION
Drawing-based communications such sketches, writ- ing, and diagrams come naturally to humans and have been used some form since ancient times. Model- ing such data is becoming an increasingly important and topical challenge area for machine learning systems aim- ing to interpret and simulate human creative expression. Such chirographic structures are challenging to interpret and generate due to their complex and unstructured na- ture. However recent progress has been strong, thanks to the advances in learning latent representations for se- quences (Bowman et al.) 2016] Graves] 2013) SSivaattaa al. 2015). In particular, with the advent of varia- tional siguencc-to-sequenee
00
=(0)
s/n]
x|
x(2)
Figure 1: Left: Discrete sequence representation. Right: More natural continuous time functional representa- tion. The gray line denotes pen-up state.
generative models (Srivas- lava al. 22055] [Bowman et al. 20166) and the collection of large-scale datasets (Gervais et al. 2020 Ha & Eck
2018 Ge al. 202T). have seen surge advance-
ments (Ha & Eck] 2018) Aksan et al. 2020) in this direction. Nevertheless, a key missing link is
the fact that drawing intrinsically continuous time, as is the resulting drawn artefact, when considered sequence rather than a raster image.
The dominant approach to model free-hand sketches, popularized by Ha Eck 2018). has been to use discrete-step recurrent network for capturing the latent process p( h n h|n - I]) with discrete step n,, and an observation model p (s[n) h(n)) )rn) to explain local structure. The large body of meth- ods built upon this have critical flaw: they ignore the very nature of the data, i.e. drawn structures are inherently continuous in time (Refer to Fig. Tp. While a few attempts have been made to use continuous-timg time constructs like Bézier curves (Das et al., 2020 2021), their inconvenient mathe- matical form affect the representational power as well as training dynamics. The reason for such an absence of continuous-time models is the lack of fundamental tools for handling continuous time

---

Published as a conference paper at ICLR 2022
data. Lately however, the introduction of Neural Ordinary Differential Equations (Chen al. 2018) followed by numerous extensions (Dupont et al. 2019 Yildiz et al. 2019 Kidger et al. 2020) have opened the possibility of building powerful models that natively represent continuous time data.
In this paper, we represent chirographic drawing data like handwriting, sketches etc. as continuous- time function s(t) and model the underlying latent process also continuous- latent function htt). Specifically, we propose a framework for capturing the derivative (potentially higher order)
h(t) of the latent process de with parameterized neural networks. At inference, a solution trajectory s(t) is computed from a given Initial Value Problem (IVP) that includes learned ODE dynamics and an initial hidden state ha - h = 0). An obvious advantage of the functional representation its arbitrary spatial resolution, i.e. we can retrieve a structure at any spatial resolution by sampling the function at appropriate rate. Moreover, with a functional representation, we can systematically upgrade the representation along time with additional properties (e.g. line thickness, color etc).
So far, parameterized ODE dynamics models have largely been treated as a "replacement for ResNets" (Massaroli et al. 2020) where the intermediate states are of little importance. While the capability of ODE models are beginning to be tested in time-series data (Kidger et al. 2020 Morrill ct al. 202T) they still remain largely unexplored. Following the introduction of Neural ODEs by Chen et al. 2018), methods have been proposed in order to regularize the solution trajectory to be as simple as possible (Kelly al., 2020 Finlay et al., 2020). However, time-series data such as chi- rography are entirely the opposite. The dynamics model, and consequently the solution trajectory should be as flexible as possible in order learn the high degree of local and global variations with time often exhibited in such data. We increase the flexibility of Neural ODE models by introducing a new class of parameterized dynamics functions. Our experiments show that this is crucial in order to model chirographic data with satisfactory fidelity. We also introduce new data augmentation method particularly geared towards ODE models and continuous-ting sequences.
Finally, we propose a deterministic autoencoder with global latent variable z that exhibits some generative model properties by virtue of inherent continuity. Our final model, SketchODE, similar to SketchRNN (Ha & Eck| 2018) in terms of high-level design but lifts it to the realm of continuous- time. Even though employed for chirographic data in this paper, our core model is the first generic continuous time Seq2Seq architecture. We explore some of the noteworthy features that differentiate SketchODE from discrete-ting -ime Seq2Seq analogoues.
An inductive bias for continuity: Discrete-time sequence models (Ha & Eck 2018) need to use their capacity to model both global structure and temporal continuity in the data. ODE models on the other hand, have strong intrinsic architectural bias towards temporal continuity. Sequences generated using a well-behaved ODE dynamics function are guaranteed to be continuous. Since they need not learn the continuity bias from scratch, ODE models are more data-efficient and are able to use majority of their capacity for modelling higher level structures. We even demonstrate that our SketchODE supports meaningful 1-shot learning when fine-tuned.
Deterministic Autoencoding with Structured Latent Space: A surprising property of Seq2Seq ODE models is the ability to perform latent space interpolation and sampling without the necessity of imposing a prior distribution p(z) on the variational posterior qa (z]s). This property is also a consequence of the latent to output mapping being continuous.
2 RELATED WORK
With the advent of recurrent neural networks, sequential time-series models have seen significant adaptation in variety of applications. Video (Srivastava et al., 2015) and natural language (Bowman) et al. 2016) were two early applications to have seen significant success. A different but important time-series modality is chirographic data like handwriting (Crraves) 2013) and sketches (Ha Fckk 2018) which gained traction only recently. Subsequent developments also address data modalities that are not flowingo like Fonts (Lopes et al. (219)) and Icons (Carlier et al. (020) which require slightly different models. Mirroring developments in natural language processing (Vaswani et al., 2017). Transformers are now replacing (Carlier et al. 2020) recurrent networks in such models.
Following the seminal work of Ha & Eck (2018), the primary representation for chirographic data has been discrete sequence of waypoints or tokens from specialized Domain Specific Languages like SVG (Scalable Vector Graphics), with minority continuing make use of standard raster graphic
2

---

Published as a conference paper at ICLR 2022
architectures (Ge al. 2021). Overall, few studies have attempted to develop better representations using specialized functional forms, with [Aksn et al. (2020) learning stroke embeddings and Das et al. (2020: 202T) directly modeling strokes as parametric Bézier curves. However, the best of our knowledge, modelling chirographic data as generic functions have not yet been tried so far.
The Neural ODE (Chen et al. 2018) provides credible tool for natively modelling continuous-time functions using dynamical systems. Since their inception, Neural ODEs have sparked a wide range of discussions and developments (Massaroli al. 2020), mostly in the theoretical domain. Neural ODEs have been extended to work with latent states (Dupont et al. (2019), to define generative mod- els (Song et al. 202I>, and have external data control (Kidger et al. 2020). While their practical training remain a challenge, significant amounts of work have been dedicated towards developing engineering techniques (Finlay et al., 2020) Grathwohl al.) 2019 Kelly et al. 2020 Poli et al., 2020) for faster convergence and better regularization. Despite flourishing theoretical interests, due to high training complexity, applied works have so far been limited few domains like Reinforce- ment Learning (Du et al. 2020). reasoning in visual question answering (Kim & Lee 2019). and
physical system identification (Lutter et al. 2019) Greydanus et al. 2019 Finzi et al.
2020 etc.
NEURAL ORDINARY DIFFERENTIAL EQUATIONS (NEURAL ODE)
Neural ODE (Chen al., 2018) provides a framework for mod- elling an inherently continuous-time process by means of its time derivative. Given a continuous-timg process s(l) IRd over a de- fined time interval [40 41). we can approximate its underlying true dynamics by learning the parameters a neural network (.). act- ing proxy for the same
2
s(t) (t, E Rd.
(1)
The time derivative of a vector-valued function, i.e. S(() defines vector field over Id which guides the evolution of the process over time. Given the initial state So s( to) and the learned set of parameters O", the process can be recovered by solving an Initial Value Problem (IVP) using any Differential Equation solver
Figure 2: Vector- depict- ing dynamics directly on s(t); modelled trajectory (red) learned from VectorMNIST data (blue).
(1)) So + /"" Fe- (1, ddl VL E [4o,4]
Given a of virying functions, one learn dynamics Fe capturing high-level concepts. The learning algorithm provided by Chen et al. (0088) makes it practical to train such models with constant memory cost (w.rt time horizon) both in forward and backward pass.
FRAMEWORK OVERVIEW
We borrow the notation of Section 3 and denote chirographic structures continuous functions of time s(t) defined within [4o,4] 41. The exact form of s(() must include the running 2D coordinates over time and may include other properties like pen-state (whether ink is visible at time () etc. In principle, one could directly model s(t) with Neural ODE as in Eq. (see Fig. 2). However, such a model would possess low representational power resulting in underfitting, and furthermore would be incapable of representing self-overlaps naturally exhibited by chirographic data.
I
4.1 NEURAL ODE AS DECODER
Augmented ODE. We first augment (Dupont et al. 2019) the original state s(l) with the underly- ing hidden process h(t) in the same time interval. Unlike discrete- time analogues like recurrent net- works which model discrete transition of the hidden state followed by an observation model p(s h), we model the "co-evolution" of the hidden process along with data using a dynamical system de- fined over the joint space a(t) := [s(t) h(tt)". Augmented ODEs (Dupont et al., 2019) improve representational power by lifting the vamilla ODE of Eq. [ to higher dimension, allowing the data trajectory to self-overlap in time. In addition, an augmented latent state h(t) provides a principled way to control the trajectory by simply solving the dynamics with different initial condition h
3

---

Published as a conference paper at ICLR 2022
s(t)
Newra ODE
5(9)
b(0)
((F)
G
100
bg
to
s(t)
t1
Figure 3: Overview of the SketchODE framework. The Neural CDE encodes input sequence s(l) to a latent vector z and a second-orde -oddrr Augmented Neural ODE decodes it as s(t).
Higher-Order Dynamics. To further increase representational power, we capture the second- order derivative ä(t) with a parameterized neural network Fe(.) like Yildiz et al. (2019). This formulation however, leads to the requirement of extra initial condition äg == alt to)
aa()) a(t)
ao ào
a Fo(+)
(2)
Traditionally, the quantities att) and a(t) termed "position" and "velocity" respectively of the dynamical system represented by ä(t) = Fe(-). Inspired by SketchRNN (Ha & Eck 2018). simplify the model by dropping the position component a(t) dependency to the dynamics function, leading to a dynamics of ä(t) = Fe(á(t),
Conditioning. Our model so far is deterministic given a particular choice of initial condition. In order to generate different samples, we need to condition the decoding on a latent vector. Thus, we introduce the following architectural changes:
1. We define a global latent vector for each sample and compute the hidden part of initial conditions by projecting it through Ldees as Hg == [b hol (z). Without loss of generality, we fix the visible part of the initial state, i.e. So and š. to be constants.
2. We further include z directly fixed parameter to the dynamics function. This is specific form of data-controlled dynamics (Massaroli et al. 2020). Please see Appendix B for implementation details.
The final second order dynamical system represented by our decoder is therefore
ä(t) Fe ft
Computing the forward pass (Eq. 2 requires the knowledge the exact value of z. The latent code can be computed by employing parameterized encoder. Similar to pacrete-timme analogues, propose to employ a non-causal model as an encoder.
4.2 NEURAL CDE AS ENCODER
We propose to Neural Controlled Differential Equations or Neural CDEs (Kidger et al.) 2020 to map given data sample y ( (t) its latent code
Neural CDEs are dynamical systems where the hidden state g(t) evolves under the control time-varying process. Given a real data sample from the dataset = (t) an initial state go g lo) is transformed by running ODE solver surrogate ODE dynamics defined as
g (t) Go (t, g) (t)
where Gp is parameterized neural network that is analogues to a Bidirectional-RNN encoder in discrete-time The latent code for the data sequence s (t) is mapping Lene of the state that fully encodes the incoming data, i.e. gı := g( f1). which given by
z Lene(gi) where gı = go f" (t. (t))..
(4)
where Go being function of time facilitates the usage of non-uniformly sampled data inference. As suggested by Kidger al. (2020), the velocities of data trajectory (() can be computed with
4

---

Published as a conference paper at ICLR 2022
any non-causal interpolation (e.g. Natural Spline curve) of the original discrete data. Alternatively, with some compromise to accuracy, crude estimates can also be computed from discrete data using forward/backkarr//eentt difference operator. Refer to Fig. 3 for overview of the architecture.
4.3 LOss FUNCTION & TRAINING
Given a ground truth trajectory (t), and the reconstructed trajectory (t) (or implied (t)) computed by executing the encoder and decoder above, we can simply minimize regression loss (w.r.t θ, & Lener Lieo) at any granularity depending on availability of data and quality requirement
(t) = d
(ts)s (t)
c (t).s) (t)
c(s
(5)
(E((ONs)
Discussion. Note that the solution trajectory computed by Eq. E entirely deterministic given the latent code. However, the end-to-end continuity of the formulation allows discovering structured latent space by default. The mapping from the input (()) to latent code z, and z to predicted trajectory (( (t) is continuous and smooth as long as the dynamics functions governing the encoder CDE and decoder ODE are lipschitz continuous. Consequently, infinitesimally small perturbation in latent space leads to an infinitesimally small change in the predicted trajectory. This property is important because it enforces structure in the latent space, enables latent space interpolations, and can be exploited to sample new data - as we will show in Section 5
stochastic generative model (Kingma & Welling 2014 Ha & Eck, 2018) by assuming standard isotropic Gaussian prior on the latent code and parameters of the dynamics. Please refer to Appendix A for the exact factorization of the joint distribution and the resulting ELBO objective Cprob. However, we focus on the deterministic variant in the main paper in order to investigate the unique properties it exhibits, as mentioned in Section [ This also has the benefit of leading to relatively simple learning dynamics due to the absence of VAE-style KL-divergence loss (Kingma & Welling, 2014), which known to notoriously challenging to optimize and require specialized engineering tricks (Higgins et al. 2017 Bowman al. 2016).
Our model also be instantiated
4.4
DATA FORMAT
Our model does not make any assumption on the exact form of s(t). However, we propose two practical data format which might be useful for different applications and lead to different trade-offs between computational cost and representation power.
Full-sequence format: s(t) is comprised of two components, the position of the pen x(t) and the state of the pen y() E {0, which denotes whether the ink should be rendered or not. Following Section 4.1 we fix Xo = 0 and yn = 0. Please note that y() is different to the traditional "pen- change" state popularized by Ha Eck 2018 which denotes whether the pen switched its state.
Multi-stroke format: Alternatively, we may represent each sample as set of constituent strokes (t), (²),......) ... where x : [ y) E R², each which represented as continuous time functions. In order to incorporate such data format, slightly modified version of the model is required. For CDE based encoder, we encode every stroke x* (t) individually using Eq. with initial state set an updated version of the last state from previous stroke with simple transformation. similar approach is followed for the decoder ODE with the end of sample decided using a binary flag predicted from last state of stroke. Please refer Appendix for detailed description.
4.5 A FLEXIBLE FAMILY OF ODE DYNAMICS
A vital component of the model with regard to capturing chirographic data properly is the family of parameterized dynamics Fe and Gz. So far, the dynamics functions have been chosen to be MLPs (Chen et al. 2018 Massaroli et al. 2020) with standard activation functions like (eLU(-), Tanh(-) and Sigmoid (.), even for time-series data (Kidger et al., 2020 Yildiz et al. 2019). This class of dynamics functions
restricted in terms of the complexity of trajectories they model and not flexible enough to capture high frequency temporal changes (c.g. multiple overlaps in short time,
sharp edges etc.) in the data trajectories. Inspired by SIREN (Sitzmann et al. 2020), which intro-
5

---

Published as a conference paper at ICLR 2022
Tanh
Sigmo'd
RelU
Cos, 0.5.5
Cos, 1.0
Cos, 2.0
0= 0.1
0.15
0.2
Vector Feelas
Trajectoriss
Figure 4: Left: 2D vector fields induced by MLPs (with random weights) with different activation functions. A few randomly sampled solution trajectories are shown below the corresponding vector fields. Right: Original data and augmented versions with different strengths of Perlin Noise.
duced way to increase spatial frequency content, we propose to periodic activation functions in order to increase the temporal frequency content of solution trajectories. We use sinusoidal func- tions, i.c. SIN(W =²) or Cos(w .r) as pointwise non-linearities. w a parameter of the activation function and controls the extent of flexibility of the dynamics functiio. w can be treated as hyperpa- rameter or can be learned individually for each layer. Fig. (left) visualizes the dynamics functions induced by MLPs with different activation functions as well as the effect of w on periodic activation. By controlling w. is possible to reduce the frequeney content of the solution trajectories under the model, leading to an "abstraction effect" (see Sec. 5.5
4.6 CONTINUOUS DATA AUGMENTATION
Data augmentation is performed for increasing the effective number of samples by creating synthetic ones from real data. It aims create data cloud around the real samples, helping the model discover local manifolds. Discrete time chirographic models are trained primarily by augmenting sequential data with independent Gaussian noise on its constituent points (Ha & Eck (018). We propose new data augmentation strategy specifically fit for continuous time sequences. We use Perlin Noise (Perlin| 1985 2002), a popular continuous noise algorithm heavily used in Procedural Generation (Lagae et al.) 2010), to sample continuous function p(t) E R in the same time interval as a given st.). The noise function is inherently continuous but can be sampled at any necessary granularity. The noise function is then centered (by subtracting its center of gravity) and added to coordinates the original sample. Refer to Fig. (right) for visual examples. The simplest way to sample an augmented version is by adding
(p(t) PCG) to the constituent strokes of s(t). where
(tt) = (erlin2D(s(t)) + + E) Perlin2D(s(()) + Ey + +e)
and PCG = = p(t)dt
The Perlin2D(-) is the original 2D Perlin Noise function in two dimension, (ears eyy] and are coor- dinate and sample level seeds to simulate pseudo-random behavior. a is the strength of the noise.
5 OBSERVATIONS & EXPERIMENTS
5.1 DATASETS& & SETUP
We experimented with three chirographic datasets to evaluate our model and compare to traditional discrete time models. We use 80% 20% train-test split for each.
Vecto: We collected small scale dataset of vectorized MNIST digits, i.e. handwritten to 9. We recorded dense points along the trajectory to capture its functional form with least degree of error. A total of 1000 instances of MNIST digits were collected. The dataset then expanded synthetically by augmenting the real data the with Perlin Noise, explained in Section 4.6
Quick Released by Ha & Eck (2018). Quick, Draw! is the largest collection of publicly available free-hand doodling dataset. These doodles are casual representations of familiar concepts drawn within a short amount time, and are diverse and free-flowing in nature. We use a few cate- gories (face, fish, flower) in order to compare model against traditional discrete time alternatives.

---

Published as a conference paper at ICLR 2022
DiDi Datase 2 Released by Gervais et al. 2020), Digital Ink Diagram dataset is large collection of synthetically generated flowcharts comprising of different components connected by arrows and lines. It has previously been used for modelling compositional relations by AAsan et al. (2020). We use the "unlabeled" version of DiDi dataset which doesn't tt have labels drawn on the diagrams.
Implementation Details
To train framework in practice, we require discretized but possibly non-uniformly sampled ground-truth data along time. For the full-sequence format, each sample in the dataset is represented by an N length sequence each including pen-states. For multi-stroke, each data sample may consists of multiple strokes each represented as a discrete coordinate sequence. For computational and representational convenience, we re-scale the time-range for both sull-seueence and every stroke in multi- stroke format to [0, T] with T being a hyperparameter. For time-sensitive applications, time can be left as it is in the raw data and can be regressed from the latent vector (full-sequence format) or initial decoder state (multi-stroke format). Please refer to Appendix for a full description of the data structures, resampling and batching mechanism used practice.
Computing the latent code using CDE based encoder requires functional form s(t) at training and inference. We use Natural Spline as suggested by Kidger et al. (2020) to construct a spline function from discrete data in order to compute the time-derivative in Eq. 4 For both encoder and decoder, the forward passes involve computing the ODE solution using a black-box ODE solver. Instead of a more accurate adaptive solver, use RK4 (The classic Runge- Kutta method) it strikes a perfect balance between computation efficiency and solution accuracy.
The pointwise regression loss L in Eq. 5 is chosen to be Huber Loss or Smooth L1 loss (Girshick 2015) and the summation is computed for a fixed sequence of uniform time points with granularity Gsng for full-sequence format and Gstroke for multi-stroke format. The granularity does not affect memory consumption as both forward and backward pass of Neural ODE models are constant- memory wrt the number of points be evaluated. Due to complex nature of data, Quick, Draw! and DiDi are trained using multi-stroke model with Gerokk = 30; otherwise Ge eeey 75, Gstroke = 25. The dimensionality of the hidden state CDE/ODE is denoted d and the latent dimension as 1. For experimentation, we choose d = 64, 1 = 48 for VectorMNIST, d = 96, I = 84 for DiDi and 144, = 96 for Quick, Draw!. Dynamics networks (for both CDE and ODE) are MLPs with few (2 for full-sequence, 1 for multi-stroke) hidden layers of size [1.2d]. For a fair comparison with discrete time models, we use an LSTM-LSTM Seq2Seq model with same hidden dimension. We empirically found the best value of the upper limit of time range to be T 5. All models are trained with AdamW (Loshchilov & Hutter 2019) optimizer, learning rate of x 10 for RNN and 3 x 10T 3 for CDE-ODE, annealed using cosine scheduler (100 epoch period) with decreasing (by factor of 4) amplitude. We also used 10-2 as weight decay (1.2 penalty) regularizer on the weight of the dynamics networks and gradients are clipped magnitude of 0).1..
5.2 RECONSTRUCTION & GENERATION TASKS
We first evaluate our framework for the tra- ditional reconstruction task and compare with discrete time RNN model. Reconstruction per- formance is an indicator of a model's ability to faithfully capture an input by means of compact latent code. We compare our deter- ministic SketchODE model with an RNN-RNN Seq2Seq model trained for deterministic recon- struction. We also train a recognition CNN on the train data and apply it to the rasterized re- construction of test data as a semantic measure of reconstruction fidelity. We compute test loss and recognition accuracy for both model on all datasets. The results (in Table. [ show that SketchODE performs comparably to standard RNN-RNN Seq2Seq models. While reconstruction loss is slightly worse, SketchODE retains recognizable visual semantics. This confirms that continuous time mod- els have a strong inductive bias towards semantic content rather than exact point-wise reconstruction. We also compare with two related alternatives: BézierSketch (Das et al. 2020) and CoSE (Aksan et al. 2020) trained deterministically.
Quick, Draw!
VectoeMNNIST
DiDi
Loss
Loss
56
Lass
$6 acc.
SkeichODE
0.0058
33.80
0.0016
98.41
0.0024
56.22
RNN-RNN
0.0048
94.20
0.0011
98.48
0.0220
96.24
COSE
0.00334
95.27
0.0007
98.02
965.31
MezierSketch
0.0121
82.12
0.0034
96.04
0.0094
91.46
Table 1: Autoencoding loss and recognition accu- racy of reconstructed input between SketchODE and traditional models.
http://sthhooooooooeeeee/////eee dataset
7

---

Published as a conference paper at ICLR 2022
5
A
Veciate 57
222
2.0
5
5
A
0.8
6.8.
5
3
0.6
3
1
0.4
6.4
7 0.0-
Periddi
Perietic
Vivill
5.0
Subeet (S) training data
30
300
Figure 5: Left: Conditional sampling in SketchODE vs discrete RNN-RNN vs CoSE. Middle: Ab- lation study to validate the use of periodic activations & Perlin noise augmentation in SketchODE. Right: SketchODE provides more data efficient learning than an RNN-RNN analogue.
We also validated our design choices of periodic activations and continuous noise augmentation for reconstruction task with an ablation study shown in Figure. (middle).
The generative model trained with an extra ppob (as described in Section 4.3 as loss generate plausible data even unconditionally (Refer to Appendix [A). However, a special property of continu- ous time Seq2Seq models like ours is conditional sampling from a deterministicaally trained network. Since the mapping from latent code to output is defined by an ODE solution trajectory, infinitesi- mal perturbation of latent codes leads to similar perturbation in the data, thus preserving semantic structure. For the discrete counterpart i.e. RNN Seq2Seq models, the sampling-based autoregressive nature of inference makes the latent output map discontinuous. For CoSE (Aksan et al., 2020), even if the the individual strokes remain continuous, their geometry and relational structures (autore- gressive) break down. We deterministically encode input data into latent code and sample from (22, oI). Figure shows qualitative results for conditional sampling with 0.05. Clearly, the autoregressive models (i.e. RNN and CoSE) failed to retain both smoothness and semantics, whereas samples from SketchODE are diverse, visually pleasing and meaningful.
5.3 DATA EFFICIENCY
Temporally continuous models are data efficient for chirographic representation due their strong inductive bias towards continuous data. The implicit distribution defined by our SketchODE model has support only over possible continuous trajectories. Discrete time sequence models on the other hand, define distribution over all possible trajectories and hence are required to "learn" the bias (over continuous trajectories) from data itself. To illustrate this we train both RNN-RNN and our SketchODE model on varying amounts of data. Dimensionality of state/hidden vectors are kept to be same for fair comparison. Fig. 5 (right) shows the test reconstruction error as a function of training data. We can see that SketchODE has particular advantage in the low data regime.
5.4 INTERPOLATION & ANIMATION
Continuing the discussion about the strong inductive bias toward continuous data, SketchODE can- not generate temporal discontinuity for any latent vector z. This leads to the property that smooth interpolation in latent codes leads to smooth interpolation the data. This is a noteworthy properly for deterministic autoencoder like SketchODE as it is not true for traditional RNN-RNN models. To illustrate this we train SketchODE and extract latent codes 2g and Z1 for two data samples s(1) and s2)), We interpolate between latent codes and produce Zinterp = Zu a Z1 * (1 a)) and decode Zinterp for several values of a [0, 1]. Fig, 6 (left, right) illustrate interpolation in VectorMNIST, Quic and DiDi datasets. More examples are shown in Appendix E
One-shot interpolation between unseen classes Existing latent models like (Ha & Eck 2018) need to be trained on substantial amounts data in order to learn the manifold required for latent- space exploration. A unique capability of SketchODE is the ability to perform interpolation after 1-shot learning of new categories. This is due to the strong inductive bias of continuity that must be learned from scratch in discrete-time analogues. We demonstrate this by creating two extra handwritten samples "A" and "B", and fine-tune our trained VectorMNIST model for approx 1k iterations. We are then able to interpolate between novel letters shown in Fig. 6 middle).
8

---

Published as a conference paper at ICLR 2022
5 y T S 8
8
5
B 3
A
2 2 4 4 f p
8
4
1 4 D O O
$ B
B
D
D
B
D
55 P C O
<
Figure 6: Illustration of SketchODE latent space interpolation and animation. Left: Interpolation + in VectorMNIST. Middle: SketchODE(top) supports interpolation be- tween novel categories (e.g.. A + B and B + D) unseen during training (on VectorMNIST), SketchRNN(bottom) fails to do this. Right: Interpolation examples QuickDraw and DiDi datasets.
+
5.5 ABSTRACTION EFFECT
Strong inductive bias towards temporal continuity enables another capability which we term the "ab- straction effect". In the context of sketch, "abstraction" (Berger et al. [ Muhammad et al. 2019) refers to removing details from the data while keeping as much semantic content as possible. Our proposed continuous SketchODE model a perfect fit for creating such abstraction. As discussed before, SketchODE can almost the entirety of its capacity for modelling the semantic content since temporal continuity comes for free. We observed that by reducing the capacity of the decoder, systematically control the amount of detail that specific choice of dynamics family, an easy way to do so is by controlling the frequency of periodic activation functions. We use SIN(-) as activation due to the fact that w + linear activation) and hence the model looses most of its capacity. Figure 7 shows that as we reduce activation function frequency, high frequency content is removed from the data the reconstruction.
be captured by the model. Due to our
SIN(W x²) + (i.e.
Docressingg tres Horeazer Stattoctoo
5
5
3
3
7
Valldaton Error
5
3 33
7
2
01
5
3
3
2
7
2
3
5
Figure 7: Illustration of SketchODE abstraction effect by varying activation frequency w. Decreas- ing the value of w, leads to increasing abstraction/reddeed high-frequency detail.
6 CONCLUSIONS
In this paper, we proposed fundamentally new paradigm for representing chirographic drawing data like handwriting, diagrams, and sketches. The SketchODE framework and its unique functional data representation posses several qualities that its discrete-time analogues do not: generation of inherently scalable rebitrary-resllution images, data efficient learning due to a built-in inductive bias towards continuity, ability to conditionally sample and perform 1-shot interpolation with a deterministically trained model. One drawback however the computational complexity (approx 4-5x more computation time than discrete counterparts) of training which require computing higher order derivatives in both forward and backward pass. The constant-memory (w.nt time horizon) nature of the training algorithm however, allows packing more samples in batch for the same memory footprint. Despite computational challenges, this work may provide a conceptual primitive for modelling more complex but constrained structures (c.g. fonts, icon and vector arts). More generally, this is the first continuous time Seq2Seq model, which may find application in diverse domains beyond chirography, like finance, audio video, which are inherently continuous in nature.

---

Published as a conference paper at ICLR 2022
REFERENCES
Emre Aksan, Thomas Deselaers, Andrea Tagliasacchi, and Otmar Hilliges. Cose: Compositional stroke embeddings. NeurIPS, 2020.
Itamar Berger, Ariel Shamir, Moshe Mahler, Elizabeth Carter, and Jessica Hodgins. Style and ab- straction in portrait sketching. ACM Trans. Graph., 2013.
Samuel Bowman, Luke Vilnis, Oriol Vinyals, Andrew Dai, Rafal Jozefowiez, and Samy Bengio. Generating sentences from continuous space. In CoNLL, 2016.
Alexandre Carlier, Martin Danelljan, Alexandre Alahi, and Radu Timofte. Deepsvg: A hierarchical generative network for vector graphics animation. NeurIPS, 2020.
Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. In NeurlPS, 2018.
Ayan Das, Yongxin Yang. Timothy Hospedales, Tao Xiang, and Yi-Zhe Song, Béziersketch: A generative model for scalable vector sketches. In ECCV, 2020.
Ayan Das, Yongxin Yang. Timothy M. Hospedales, Tao Xiang, and Yi-Zhe Song. Cloud2curve: Generation and vectorization of parametric sketches. In CVPR, 2021.
David H. Douglas and T. Peucker. Algorithms for the reduction of the number of points required to represent digitized line or its caricature. Cartographica: The International Journal for Geo- graphic Information and Geovisualization, 1973.
Jianzhun Du, Joseph Futoma, and Finale Doshi-Velez. Model-based reinforcement learning for semi-markov decision processes with neural odes. In NeurlPS, 2020.
Emilien Dupont, Arnaud Doucet, and Yee Whye Teh. Augmented neural odes. In NeurlPS, 2019.
Chris Finlay, Jörn-Henrik Jacobsen, Levon Nurbekyan, and Adam M. Oberman. How train your neural ODE: the world of jacobian and kinetic regularization. In ICML, volume 119, 2020.
Mare Finzi, Ke Alexander Wang, and Andrew Gordon Wilson. Simplifying hamiltonian and la- grangian neural networks via explicit constraints. In NeurIPS, 2020.
Songwei Ge, Vedanuj Goswami, Larry Zitnick, and Devi Parikh. Creative sketch generation. In
ICLR, 2021.
Philippe Gervais, Thomas Deselaers, Emre Aksan, and Otmar Hilliges. The DIDI dataset: Digital ink diagram data. CoRR, 2020.
Ross Girshick. Fast r-cnn. In International Conference on Computer Vision (ICCV), 2015.
Will Grathwohl, Ricky T. Q. Chen, Jesse Bettencourt, Ilya Sutskever, and David Duvenaud. FFJORD: free-form continuous dynamics for scalable reversible generative models. In ICLR, 2019.
Alex Graves. Generating sequences with recurrent neural networks. arXiv:1308.0850 2013.
arXiv preprint
Samuel Greydanus, Misko Dzamba, and Jason Yosinski. Hamiltonian neural networks. In NeurlPS, 2019.
David Ha and Douglas Eck. A neural representation of sketch drawings. In ICLR, 2018.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In ICLR, 2017.
Jacob Kelly, Jesse Bettencourt. Matthew J. Johnson, and David Duvenaud. Learning differential equations that easy to solve. In NewrlPS, 2020.
10

---

Published as a conference paper at ICLR 2022
Patrick Kidger, James Morrill, James Foster, and Terry J. Lyons. Neural controlled differential equations for irregular time series. In NeurlPS, 2020.
Wonjae Kim and Yoonho Lee. Learning dynamics of attention: Human prior for interpretable ma- chine reasoning. In NeurIPS, 2019.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. ICLR, 2014.
Ares Lagae, Sylvain Lefebvre, Robert L.. Cook, Tony DeRose, George Drettakis, David S. Ebert, John P. Lewis, Ken Perlin, and Matthias Zwicker. A survey of procedural noise functions. Comput. Graph. Forum, 29, 2010.
Raphael Gontijo Lopes, David Ha, Douglas Eck, and Jonathon Shlens. A learned representation for scalable vector graphics. In ICCV, 2019.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
Michael Lutter, Christian Ritter, and Jan Peters. Deep lagrangian networks: Using physics as model prior for deep learning, In ICLR, 2019.
Stefano Massaroli, Michael Poli, Jinkyoo Park, Atsushi Yamashita, and Hajime Asama. Dissecting neural odes. In NeurlPS, 2020.
James Morrill, Cristopher Salvi, Patrick Kidger, and James Foster. Neural rough differential equa- tions for long time series. In ICML, 2021.
Umar Riaz Muhammad, Yongxin Yang, Timothy M Hospedales, Tao Xiang, and Yi-Zhe Song. Goal-driven sequential data abstraction. In ICCV, 2019.
Ken Perlin. An image synthesizer. ACM SIGGRAPH, 19, 1985.
Ken Perlin. Improving noise. ACM SIGGRAPH, 21, 2002.
Michael Poli, Stefano Massaroli, Atsushi Yamashita, Hajime Asama, and Jinkyoo Park. Hyper- solvers: Toward fast continuous-depth models. In NeurlPS, 2020.
Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows. In ICML, 2015.
Vincent Sitzmann, Julien N. P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wet- zstein. Implicit neural representations with periodic activation functions. In NeurlPS, 2020.
Yang Song. Jascha Sohl-Dickstedi Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLR, 2021.
Nitish Srivastava, Elman Mansimov, and Ruslan Salakhudinov. Unsupervised learning of video representations using Istms. In ICML, 2015.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017.
Cagatay Yildiz, Markus Heinonen, and Harri Lähdesmäki. ODE2VAE: deep generative second order odes with bayesian neural networks. In NeurIPS, 2019.
11

---

Published as a conference paper at ICLR 2022
A GENERATIVE LOss FUNCTION
In order to provide extra flexibility to model variations, we can define explicit density over decoded trajectories. We enforce distributions g(z)s(0) and (()) over the latent code and the pa- on
N
θ
rameters of the dynamics respectively. With a conditional density
treat the ODE trajectory Continuous Normalizing Flow or CNF (Chen et al. 2018) that transforms the induced den- sity (HH (2) through continuous mapping. We sample from the conditional and prior by means reparameterization (Kingma & Welling 2014) and compute the full loss which includes another component Lprob along with Liit accounting for the probabilistic form.
Hn
B
Figure 8: The generative model as probabilistic graph.
We first define the generative model g(s) comprised of global latent vector with prior p(z), the stochastic ODE parameters with prior p(@) on all its parameters
(i.e. ODE dynamics being Bayesian Network) and the initial hidden state Hg- The variational distribution is assumed to be factorized as
g gH, Ho. z, O|s) (()) (z(s) ((Hl)z) HoH θ)
Please refer Fig. for visual illustration of the dependency structure. The notations and H (without time suffix) denote random variables that describe continuous time sequences within the time range (4o, 11.. Therefore, the Evidence Lower Bound objective denoted as Lprab is given by
(2, 0;s()) -KL (2,, H. O|s( H, H. + E.ce.H, (eee))))Ee...llleee [og 7(s
H, O)
(6)
For deterministic mappings, we assume uninformative priors, i.e. the densities are one. So, p(Holz) Ho, θ) Also, we are defining any explicit observation model, i.e.
((|HH, 9) 1. The following simplification can be done to Eq. based on assumptions
Cprob KL (g(z, (1s) H, θ)]
log (((s)) (())
(H|2, 1 θ)
p(z)
p())
p(H|2,
log (z(ss)
log ((0)
E
Ey((e)
log feNz (H|Ho, Q) () (Hulz)]
p(z)
p(0)
KL Divergence
KL Divergence
KL [g(z|s))l((() KL (a())
E (x)s)) erag(e Ho= (x) H ODE
lor genr (HIHHo, ()oxr(Holzz)
Restricting Ldee be invertible, the log-density of the induced q (Holz)) can be computed using change of variable formula as shown in Rezende & Mohamed (015))
log qsr (Holz) log ((x|s) log del Ö
Oz
The log density of the hidden trajectory be computed using the instantaneous density formula for Continuous Normalizing Flow (CNF) derived by Yildiz et al. (2019)
log yENE HH]Ho, θ)
Σ log gNr (Holz)
dFf
dá
Both q(z) -) and p(O) are modelled with isotropic Gaussians. With a reparameterized sample from both, Eq. 6 can be computed and added to the regression loss. Note that all hidden states in above equation can be replaced with its augmented version since visible states are deterministic given other unobserved quantities.
12

---

Published as a conference paper at ICLR 2022
5
8
20
0
9
6
1
2
8 294
3
3
3
9
954
5
7
1
1
9
7
3
4
1
4
7
6
5
4
6
3
Figure 9: Original data from VectorMNIST on the left and unconditional samples the right. Just like any VAE-based generative model, reconstruction is traded-off by posterior-matching objective. Sometimes, the model generates samples that are blend of two categories (dotted boxed; blend of and 5), or samples that are entirely unrecognizable (solid boxed).
We experimented with basic version of the generative model on VectorMNIST dataset and samples unconditional samples from the model. Please refer to figure for visual examples.
B DATA-CONTROLLED DYNAMICS
dynamical system can be controlled by a parameter that not "evolve" over time but can be dif- ferentiated with. Such design loosely inspired by SketchRNN (Ha & Eck 2018) which concate- nates decoder inputs with the global latent vector in order to prevent the recurrence from forgetting the initial state. With the dynamics function (á, 2,), 2),
easily achieve this in practice by
solving the following dynamical system
a(I) Fo
à f
T ;
(a, 0
The fixed zero vector as second derivative prevents it from co-evolving along with å Since is part of the state, Adjoint Backpropagation (Chen et al. 2018) naturally allows easy computation of the partial derivative.
C MULTI-STROKE FORMAT
bg
80
G
G$
Fe
Fe
AAD
A
100
(t)
(t)
Figure 10: SketchODE variant for multi-stroke format.
Unlike full-sequence format, we use the architecture defined the main paper for processing one single stroke at time while sharing the parameters across different strokes. Refer to Fig. 10 for concise diagram.
For CDE based encoder, sequence of strokes {x² (²),, is processed individually with CDE with initial state g := g* = ty) and final state gf := g" (t = to). The time range for each stroke, if not available in dataset, can be fixed as hyperparameter. Initial state for the 4th stroke can be computed by projecting the last state of (k 1) stroke with parametric transform (e.g. ML.P)
13

---

Published as a conference paper at ICLR 2022
g [(g( where gi- gb- + / (t, x=) (t).
The initial state of the very first stroke is always fixed to constant, e.g. 8i 0.
For ODE based decoder, similar strategy is followed with the exception that the initial state of the first stroke is set to Ao where $o So = O and Ho (z). The initial state of the kch stroke produced by transforming the last state of (k: - 1)th stroke with another parametric transform (e.g. MLP)
H = V(AA-11 1) where
[(t)) Fe
Af A k-1
dt
Please note that for every stroke, the initial position and velocity is fixed to zero. Such strategy leads to the ODE dynamics learning shared stroke model. follows earlier works like Das et al. (2020) 2021); Aksan et al. (2020) and require starting position v for every stroke be predicted separately, which can be regressed during training. Also, an end-of-sample flag is necessary for terminating the inference which can be achieved by defining random variable r* E {0, 1] as whether the hith stroke is the last stroke and predicting the probability of it being one
p(r = 1[A 8) = (Ro
D PRACTICAL DATA STRUCTURES & MINI-BATCHING
xfO
y(3
7.1
L4
7.3
42
6.9
4.2
6.9
3.5
833
34
8-1
(0)
(1
3.0
8.0
3.0
7.9
3.0
7.8
Full Sequznoe
Mulki - stroke
Mini Batching
Figure 11: Visual illustration of data formats, resampling and mini- Mini-batching strategy.
Samples full-sequence data format represented by a length N sequence [(t,,(( = 11)
where E [0, T are non-uniform time-points. For multi-stroke format, each data sample may consists of multiple strokes each represented as a discrete coordinate sequence. The end coordinate of the previous stroke and starting coordinate of the following stroke are connected via straight line for all time points on it; otherwise y(t 1²) O. Please refer to Fig. I] for visual illustration. For more time-sensitive applications, the time-range can be left as it is in the raw data and can be regressed from the latent vector (full- ffull-squeecc format) or initial decoder state (multi-stroke format). At inference, given latent code, the decoder ODE can be solved using the predicted time-range.
and yt (5) is
In practice, every sample in the dataset needs to have format shown in Fig. 11 For full sequence format, every stroke including the pen-up strokes (i.e. joining line between previous stroke-ending and next stroke-begining) are kept in order. The pen-up stroke is shown in gray and the correspond- ing y(t) is set to one. The multi-stroke format do not require any y()). All samples are spatially scaled down inside the unit circle, i.c. ||x(t)]l2 1 Vt E 10, Tj. For datasets with high temporal sampling rate (e.g. DiDi and Quick, Draw!), we reduce it by Ramer Daaeer- Douglas-Peuckee Peucker algorithm (Douglas & Peucker 1973) preprocessing step.
14

---

Published as a conference paper at ICLR 2022
Efficient training of Neural ODE/CDE models further require mini-batching, which is different than traditional models. For samples with different number time-points, we create a common sequence of time-points by umion-ing individual time-point sequences. An ODE solution trajectory is then evaluated at the time points the union and masking is done to discard irrelevant time points for more efficiency, sequence of uniform time-points, This is achieved by linear interpolation of points along time. Please refer Fig. for visual illustration.
each sample. For
re-sample each data in mini-hatch at fixed
E VISUAL SAMPLES
5
5
3
9
9
9
9
9
9
9
4
4
4
4
4
22
9
9
9
R
2
2
4
4
4
4
888
8
8
8
7
2
2
8
8
8
8
3
3
3
3
5
5
5
3
3
3
3
3
3
3
}
5)
S
S
S
5
5
5
4
4
$
5
5
5
6
L
6
4
4
7
10
7
1
5
3
3
3
7
4
3
3
5
5
F
>
J
5
5
5
F
Figure 12: Interpolation on latent space by cyclic walk through samples of different categories. SketchODE (above) is far better with smooth animatic interpolation while RNN-RNN not only fails to faithfully reconstruct but also suffers from severe mode collapse.
15

---

Published as a conference paper at ICLR 2022
H
F
8
0
Figure 13: Some examples of conditional sampling (above) and animatic interpolation (below).
For conditional sampling, the samples conditioned the sketch in the first row. Each of the interpolation examples is to be viewed left to right.
16


Reviews:
[{"review_id": "rjlYSv2tCjM", "reviewer": "Reviewer_Q3GY", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "+ new domain using Neural ODE\n+ well written\n+ well-thought domain-specific implementation tricks to adopt Neural ODE\n+ evaluated with multiple drawing datasets\n- limited domain experiment\n- incomplete ablation study\n- heavy dependence on the original Neural ODE work\n\nThe paper presented a list of implementation tricks to improve the performance of their Neural-ODE-enabled sequential model. The ideas of using the Perlin Noise and periodic activation functions are up-to-date and show their effectiveness to a certain degree.\n\nAlso I greatly appreciate the authors providing the details for working with the Multi-stroke format data, not just with Full-sequence format. \n\nThe paper is very well written, but it was difficult to assess the level of novelty. The authors claim that this is a new continuous-time sequential model, but they only tested their model with drawing datasets, that seems to me that there is a little bit of overstatement. This paper could be improved if the authors provide experimental results in other sequential domains, such as speech. Also, adding to that, it would be most appreciative if the authors could add a list of main contributions when this is compared against Neural ODE, as many ideas are dependent on this previous paper and it made me difficult to easily evaluate the level of contributions.\n\nFinally, it could have been better if other new methods that work with sequential data, such as transformer-enabled models, are added to the baseline. I found it very odd that the authors did not include any of those methods for comparison.", "summary_of_the_paper": "This paper presents a novel approach to represent chirographic drawing data with Neural ODE and demonstrates its effectiveness by comparing with the standard baseline Seq2Seq model.", "summary_of_the_review": "Overall, the paper is very well-written and shows fascinating results with continuous-time seq2seq model. \nHowever there are still some missing components in the paper, and more clarification needed to distinguish this method from the Neural ODE work. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "nsKDN0sGzs-", "reviewer": "Reviewer_zrrF", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I think this is a strong paper which introduces an interesting model and demonstrates various compelling properties of this model through a series of experiments on several handwritten datasets. The main strengths and weaknesses (in my eyes) are described below.\n\n**Strengths**:\n- I think the main strength of the paper is the demonstration of the various compelling properties that arise from modeling the handwritten data continuously. In particular, I really like the experiments showing that by simply adding noise to the latent vector we can generate conditional samples from the model, due to the continuous nature of the setup (which would break down in the autoregressive RNN case). The latent interpolation experiments are also very nice. The one shot learning capabilities demonstrated on VectorMNIST are also compelling.\n- The model and ideas are very well motivated. Handwriting and drawing are inherently continuous, so it makes sense to model this data continuously. The authors do a good job of demonstrating this, both by contrasting it with previous work and by showing through experiment that this works well.\n- The paper is generally well written and the figures are great.\n- While the model is somewhat complicated and has lots of moving parts, each part is generally quite well motivated.\n- The experiments are generally thorough and well done (on 3 fairly different datasets). The visualizations of the results are great. However, as discussed in the weaknesses section, it would be nice to have more samples/examples from the model (particularly on the more complicated datasets), as well as having error bars.\n\n**Weaknesses**:\n- It would be nice to have more samples and results from the model in the appendix. The majority of the results both in the main paper and appendix are shown on VectorMNIST. It would be nice to have e.g. more examples of latent interpolations and conditional sampling on Quick Draw and DiDi. At this point, since there are so few examples they can feel cherrypicked.\n- The authors do not discuss the large amount of relevant work that has been done around using neural ODEs for physics (see Hamiltonian neural networks and Lagrangian neural networks and the numerous follow up works since then). Sentences like “there has been little applied work using neural ODEs” and “parameterized ODE dynamics models have largely been treated as a replacement for ResNets where the intermediate states are of little importance” are misleading. As far as I can tell, the authors do not cite a single neural ODE + physics paper which both use the intermediate states and are applications of neural ODEs.\n- The authors overclaim their contributions in certain parts of the paper. This is already a great paper, so it is not necessary to do this. For example, “we increase the flexibility of Neural ODE models by introducing a new class of parameterized dynamics functions”. As far as I can tell the authors simply replace tanh/relu MLPs with SIREN (a pre-existing model), which effectively corresponds to changing an activation function. The properties of SIREN models are already well described in other papers, so I think it is a stretch to call this “introducing a new class of parameterized dynamics functions”. In addition sentences like “Learning meaningful representations for chirographic drawing data such as sketches, handwriting, and flowcharts is a gateway for understanding and emulating human creative expression” seem unnecessarily grandiose and don’t provide much information.\n- It would be nice to have a more thorough discussion of limitations. As far as I can tell, the only discussion of limitations is one sentence in the conclusion mentioning the computational complexity of the method. It would be nice to have some more thorough discussion of this or ideally a quantitative analysis of how much slower/expensive it is.\n- On page 1, the notation h[n] and s[n] is used without being introduced. It only becomes clear what this is later in the paper.\n- There are quite a few typos in the paper. I think it would be worth going over the paper another time to clean these up.\n\n**(Some) Typos**:\n- Page 3, top: “as generic functions have not yet been” -> “as generic functions have not yet been”\n- Page 3, top: “Neural ODE framework” -> “The neural ODE framework”\n- Page 3, bottom: “with Neural ODE as in Eq 1” -> “With a Neural ODE as in Eq 1”\n- Page 4, middle: “computing the forward pass require” ->  “computing the forward pass requires”\n- Page 4, middle: “a non-causal model as encoder” -> “a non-causal model as an encoder”\n\n*Note*: While I am quite familiar with neural ODEs, I am less familiar with the literature around learning representations for handwritten data, and have therefore put a confidence score of 3 on the review. The authors seem to provide a thorough discussion of related work for handwritten data, but it is possible that there are papers that have been missed in that area which I do not know about.\n", "summary_of_the_paper": "This paper proposes a new model, called SketchODE, for learning representations of sketches using neural ODEs. Specifically, the authors parameterize hand drawn strokes as solutions of ODEs and build an auto encoder like framework for learning the vector fields of these ODEs from data. \n\nThe decoder is modelled with a second order neural ODE acting on an augmented state (which allows for self overlapping trajectories which is necessary for handwritten data) that includes the stroke itself as well as an underlying hidden state. The decoder is made conditional by passing a latent vector to the model, which is mapped both to the initial state of the ODE as well as the dynamics function of the ODE. Varying this latent vector and solving the ODE then gives rise to different handwritten patterns.\n\nThe encoder is parameterized by a neural CDE which takes as input some ground truth trajectory and returns a new state which is mapped to the latent vector. The model is then trained end to end using a reconstruction loss between the ground truth and reconstructed trajectories.\n\nIn addition, the authors show that two additional tricks can further improve performance: using sine activation functions in the MLPs parameterizing the vector fields as well as using perlin noise to create (continuous) augmentations of the data.\n\nThe authors then discuss and demonstrate various compelling properties of their model. This includes the existence of a latent space (which allows for interpolation of handwritten data) as well as one shot learning capabilities for new classes of data.\n\nThe main contributions of the paper in my eyes are then:\n- Introducing an interesting continuous neural model of handwritten data\n- Demonstrating compelling properties of this continuous model over traditional discrete models\n- Introducing the VectorMNIST dataset, which I believe could be useful for other researchers\n- Interesting experiments on various handwritten datasets\n\n\n", "summary_of_the_review": "This paper introduces a new continuous time model for learning representations of handwritten data. The paper is well motivated and well written and demonstrates various compelling properties of the proposed model through thorough experiments. While the paper has a few minor weaknesses, I believe it deserves to be accepted.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "PZPqv7tIT97", "reviewer": "Reviewer_S7jh", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths\n- The newly proposed continuous-time generative model is novel, and several intriguing observations are made. Although the chirographic data itself might have limited application, it served as a great benchmark for testing such a continuous-time generative model.\n- The paper is clearly written and well organized overall.\n- Considerations of the application on different data domain makes the implication broader.\n\n### Weaknesses\n- The tradeoff between computational complexity and representation power induced by using Neural ODEs is not directly handled. To mitigate this, the authors might want to simply share some information about the train/prediction cost and compare them with existing methods.\n- For the sinusoidal activation functions, the frequency needs to be manually selected.\n\n### Questions\n- In the context of the multi-stroke dataset in Appendix C, what exactly are linear transformations u, v? Are they learned?\n\n### Minor typos found\n- At the end of the Introduction part: A surprising 'properly' of Seq2seq... -> property\n- Section 4.2: 'Nautral' Spline curve -> Natural\n- Section 5.1: 'Range'-Kutta method -> Runge\n", "summary_of_the_paper": "By introducing the techniques from neural ODEs on chirographic data, the authors were able to train a first continuous-time generative model. New observations that are only available on continuous-time generative models are made.", "summary_of_the_review": "This work successfully proposed the first continuous-time generative model and discovered intriguing observations. The possibility of different applications might interest a wide range of ICLR audiences, and hence I recommend accepting the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "H5GG-6iEpv", "reviewer": "Reviewer_KvGm", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors propose an approach of representing online handwriting data in the continuous space. One of the central claims made by the authors is that models that use discrete representation need to spend their learning capacity for learning both the global structure and the temporal continuity (basically, the sampling method), while the models using continuous representation can use their full capacity to represent the global structure only, thus becoming more data-efficient.\n\nAuthors represent the continuous state of the pen $s(t)$ (which can include either only coordinates, if a single stroke is encoded, or both the coordinates and the pen state, if several strokes are encoded together) as the observations following the hidden process h(t). Authors also capture the \"velocity\" of the pen tip in their model. To obtain the ground truth data for it, they fit spline curves to the existing data points.\n\nThe encoder of the proposed model used Neural CDEs and computes the hidden state by integrating the input trajectory with a kernel that is a parametrized neural network. The decoder of the proposed model uses Neural ODEs to model the velocity, which can then be turned into the recostruction of the trajectory itself. Both during encoding and decoding, the forawrd pass involves computing an ODE solution, which is done by the black-box ODE solver, using Range-Kutta method. The loss is a point-wise loss (Huber loss or smooth L1) between the points of real and recostructed trajectory, with different sampling frequencies according to the dataset. Authors provide extensive details on the training procedure.\n\nAuthors evaluate their approach on QuickDraw dataset of handwritten sketches, DiDi dataset of handwritten diagrams, and the collected small-scale dataset of vectorized MNIST digits. Authors suggest a data augmentation technique to increase the amount of training data, based on adding continuous noise to the continuous trajectory, rather than by adding independent Gaussian noise to the points.\n\nAuthors perform the following evaluations:\n* Accuracy of the trained recognition CNN on the reconstructed data from 4 models: RNN-RNN seq2seq model trained on raw points data, the one proposed by the authors (SketchODE), and two others: CoSE which encodes a discrete sequence into an implicit representation from which the points of the recostructed curve can be sampled by providing a $t\\in[0,1]$, and BezierSketch, which operates on Bezier curves parameters as inputs / outputs to the autoencoder model. CoSE achieves the highest accuracy, closely followed by RNN-RNN, and SketchODE.\n\n**Concern** The Table 1 describing the recognition accuracy of the classifier trained on the real samples, on the reconstructed samples, is meaningless without knowing what is the recognition accuracy of the said classifier on the real data. If it is, for example, lower than 93%, that means that the autoencoder models suffer from a mode collapse and are only able to generate some modes of the training data distributions, that are more recognizable, than the whole distribution in general.\n\n* Authors perform visual comparison to RNN-RNN in terms of the quality of reconstructions, data efficiency, and the ability to interpolate in the latent space. They show favourable comparisons in all cases.\n\n**Concern** Authors compare to the model that operates on the discrete sequence of points, but ignore the comparison to BezierSketch, and, more importantly, CoSE, which are able to provide the continuous reconstructions, and which would have been much fairer comparisons.\n\n**Concern** Authors talk about data efficiency, but don't mention whether they use the same data augmentation when training RNN-RNN (adding Perlin noise), or is that only applicable when training SketchODE (and, if that is the case, whether an independent Gaussian noise data augmentation is applied to RNN-RNN). Authors also don't mention the effect of the data augmentation on the performance of their model.\n\n**Concern** Authors show  the interpolation in latent space between two novel objects, and compare the quality to RNN-RNN. While this indeed indicates the smoothness of the learned latent space, but, (1) this is incomplete without comparisons to methods that allow outputting a continuous curves, and (2) interpolation in the vector space of curves would have likely produced the results similar to the ones obtained by the authors.\n\nAuthors show an interesting emergent property of their work, namely, by controlling the capacity of the model decoder, they can control the amount of \"complexity\" or \"high frequency\" in their reconstructions.\n", "summary_of_the_paper": "The paper describes a way of representing online handwriting data, typically seen as a discrete sequence, in a continuous space, using neural ODEs. The learned representation allows sampling from and interpolation in the latent space. The results in the paper are compared (mostly qualitatively) to autoencoder-learned representations based on representing data as discrete sequences, Bezier curves, and implicit surface representation from differentable geometrty.", "summary_of_the_review": "Authors propose a way of encoding online handwriting data in a continuous space representation by using Neural ODEs, and show interesting properties of the learned model. This is a novel idea interesting to wider audience.\n\nTheir main weakness of the paper, however, is the experimental section, and based on its' current content I can not recommend the paper for acceptance. \n* Most importantly, the comparisons are only done to an approach that operates purely on the discrete sequences of points, rather than the ones that, similar to the approach outlined by the authors, allow sampling a continuous trajectory as the output of their decoder (BezierSketch, CoSE), and whether the provided approach provides capabilities beyond those is an open question. Combined with the complexity of the training requiring ODE solvers, it is not clear whether the proposed approach provides any advantages.\n* The ablation study is virtually non-existent and does not allow to understand the effects of some of the decisions made by the authors: how does continuous data augmentation affect the quality of the results, rather than the validation error? What are the trade-offs between using a single-stroke representation vs multiple strokes representation? How does the multiple strokes model behave with sequences containing longer than 5 strokes, and how does a single stroke model handle long strokes?", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The SketchODE submission is a continuously-valued model for chirographic drawing data such as handwritten digits or sketches. It relies on variational sequence-to-sequence model where the latent code z is a global encoding of the drawing dynamical, and contains a neural controlled differential equation encoder to encode a discrete 2D drawing sequence s, and an augmented neural ODE decoder (conditional on the latent code z) to model both the first-order dynamics both of the drawing velocity and of the pen state (effectively modelling second order dynamics on the pen position). The model enables to sample sketches by sampling latent codes, as well as to interpolate between two latent codes, and is evaluated on VectorMNIST (a new task), QuickDraw sketches, and DiDi schematics, where it is compared to discrete RNN-based Seq2Seq and two more recent baselines.

Reviewers praised the idea of using continuously-valued Neural ODEs for drawing, compelling properties of the model for conditional generation or interpolation, the new VectorMNIST dataset, and the writing. Reviewers had some concerns: overstating the novelty and contribution to general continuous seq2seq given that the evaluation was done only on chirographic drawing tasks (Q3GY, zrrF), some experimental details such as missing ablations, examples from QuickDraw or Didi, or comparisons with transformers (Q3GY, zrrF), clarifications on computational complexity (zrrF, S7jh), situating the work with respect to applications of Neural ODEs to physics (zrrF); most of these concerns were addressed in the rebuttal. Reviewer KvGm had the most concerns about the experimental section, but has increased their score after the discussion with the authors.

There was no discussion among the reviewers, only between the authors and reviewers zrrF and KvGm. After the authors' rebuttal, the scores became 8. 8, 6 and 5, and thus I believe that the paper meets the conference acceptance bar.
Decision: Accept (Poster)