Year: 2021

Abstract:
Learning meaningful representations for chirographic drawing data such as sketches, handwriting, and flowcharts is a gateway for understanding and emulating human creative expression. Despite being inherently continuous-time data, existing works have treated these as discrete-time sequences, disregarding their true nature. In this work, we model such data as continuous-time functions and learn compact representations by virtue of Neural Ordinary Differential Equations. To this end, we introduce the first continuous-time Seq2Seq model and demonstrate some remarkable properties that set it apart from traditional discrete-time analogues. We also provide solutions for some practical challenges for such models, including introducing a family of parameterized ODE dynamics & continuoustime data augmentation particularly suitable for the task. Our models are validated on several datasets including VectorMNIST, DiDi and Quick, Draw!.

text: 1 INTRODUCTION
Drawing-based communications such as sketches, writing, and diagrams come naturally to humans and have been used in some form since ancient times. Modeling such data is becoming an increasingly important and topical challenge area for machine learning systems aiming to interpret and simulate human creative expression. Such chirographic structures are challenging to interpret and generate due to their complex and unstructured nature. However recent progress has been strong, thanks to the advances in learning latent representations for sequences (Bowman et al., 2016; Graves, 2013; Srivastava et al., 2015). In particular, with the advent of variational sequence-to-sequence generative models (Srivastava et al., 2015; Bowman et al., 2016) and the collection of largescale datasets (Gervais et al., 2020; Ha & Eck, 2018; Ge et al., 2021), we have seen a surge of advancements (Ha & Eck, 2018; Aksan et al., 2020) in this direction. Neverthe-
less, a key missing link is the fact that drawing is intrinsically continuous in time, as is the resulting drawn artefact, when considered as a sequence rather than a raster image.
The dominant approach to model free-hand sketches, popularized by Ha & Eck (2018), has been to use a discrete-step recurrent network for capturing the latent process p(h[n]|h[n− 1]) with discrete step n, and a complex observation model p (s[n] | h[n]) to explain local structure. The large body of approaches built upon this have a critical flaw: they ignore the very nature of the data, i.e. drawn structures are inherently continuous in time (Refer to Fig. 1). While a few attempts have been made to use continuous-time constructs like Bézier curves (Das et al., 2020; 2021), their inconvenient mathematical form affect the representational power as well as training dynamics. The reason for such an absence of continuous-time models is the lack of fundamental tools for handling continuous time data. Lately however, the introduction of Neural Ordinary Differential Equations by Chen et al. (2018) followed by numerous extensions (Dupont et al., 2019; Yildiz et al., 2019; Kidger et al., 2020) have opened the possibility of building powerful models that can natively represent continuous time data such as handwriting, sketch etc.
In this paper, we represent chirographic drawing data like handwriting, sketches etc. as continuoustime function s(t) and model the underlying latent process also as continuous-time latent function h(t). Specifically, we propose a framework for capturing the derivative (potentially higher order) of the latent process dKh(t)
dtK with parameterized neural networks. At inference, a solution trajectory
ŝ(t) is computed from a given Initial Value Problem (IVP) that includes a learned ODE dynamics and an initial hidden state h0 , h(t = 0). An obvious advantage of the functional representation is its arbitrary spatial resolution, i.e. we can retrieve a structure at any spatial resolution by sampling the function at appropriate rate. Moreover, with a functional representation, we can systematically upgrade the representation along time with additional properties (e.g. line thickness, color etc).
So far, parameterized ODE dynamics models have largely been treated as a “replacement for ResNets” (Massaroli et al., 2020) where the intermediate states are of little importance. While the capability of ODE models are beginning to be tested in time-series data (Kidger et al., 2020; Morrill et al., 2021), they still remain largely unexplored. Following the introduction of Neural ODEs by Chen et al. (2018), methods have been proposed in order to regularize the solution trajectory to be as simple as possible (Kelly et al., 2020; Finlay et al., 2020). However, time-series data such as chirography are entirely the opposite. The dynamics model, and consequently the solution trajectory should be as flexible as possible in order to learn the high degree of local and global variations with time often exhibited in such data. We increase the flexibility of Neural ODE models by introducing a new class of parameterized dynamics functions. Our experiments show that this is crucial in order to model chirographic data with satisfactory fidelity. We also introduce a new data augmentation method particularly geared towards ODE models and continuous-time sequences.
Finally, we propose a deterministic autoencoder with a global latent variable z that exhibits some generative model properties by virtue of inherent continuity. Our final model, SketchODE, is similar to SketchRNN (Ha & Eck, 2018) in terms of high-level design but lifts it to the realm of continuoustime. More generally, our model is the first continuous time Seq2Seq architecture. We explore some of the noteworthy features that differentiate SketchODE from discrete-time Seq2Seq analogoues.
An inductive bias for continuity: Discrete-time sequence models (Ha & Eck, 2018) need to use their capacity to model both global structure and temporal continuity in the data. ODE models on the other hand, have a strong intrinsic architectural bias towards temporal continuity. Sequences generated using a well-behaved ODE dynamics function are guaranteed to be continuous. Since they need not learn the continuity bias from scratch, ODE models are more data-efficient and are able to use majority of their capacity for modelling higher level structures. We even demonstrate that our SketchODE supports meaningful 1-shot learning when fine-tuned.
Deterministic Autoencoding with Structured Latent Space: A surprising property of Seq2Seq ODE models is the ability to perform latent space interpolation and sampling without the necessity of imposing a prior distribution p(z) on the variational posterior qφ(z|s). This property is also a consequence of the latent to output mapping being continuous.

heading: 2 RELATED WORK
text: With the advent of recurrent neural networks, sequential time-series models have seen significant adaptation in a variety of applications. Video (Srivastava et al., 2015) and natural language (Bowman et al., 2016) were two early applications to have seen significant success. A different but important time-series modality is chirographic data like handwriting (Graves, 2013) and sketches (Ha & Eck, 2018) which gained traction only recently. Subsequent developments also address data modalities that are not free-flowing, like Fonts (Lopes et al., 2019) and Icons (Carlier et al., 2020) which require slightly different models. Mirroring developments in natural language processing (Vaswani et al., 2017), Transformers are now replacing (Carlier et al., 2020) recurrent networks in such models.
Following the seminal work of Ha & Eck (2018), the primary representation for chirographic data has been discrete sequence of waypoints or tokens from specialized Domain Specific Languages like SVG (Scalable Vector Graphics), with a minority continuing to make use of standard raster graphic architectures (Ge et al., 2021). Overall, few studies have attempted to develop better representations using specialized functional forms, with Aksan et al. (2020) learning stroke embeddings and Das
et al. (2020; 2021) directly modeling strokes as parametric Bézier curves. However, to the best of our knowledge, modelling chirographic data as generic functions have not yet been tried so far.
The Neural ODE (Chen et al., 2018) framework provides a credible tool for natively modelling continuous time functions using dynamical systems. Since their inception, Neural ODEs have sparked a wide range of discussions and developments (Massaroli et al., 2020), mostly in the theoretical domain. Neural ODEs have been extended to work with latent states (Dupont et al., 2019), to define generative models (Song et al., 2021), and have external data control (Kidger et al., 2020). While practical training of ODE models remain a challenge, significant amounts of work have been dedicated towards developing engineering techniques (Finlay et al., 2020; Grathwohl et al., 2019; Kelly et al., 2020; Poli et al., 2020) for faster training convergence and better regularization. Despite flourishing theoretical interests, due to high training complexity, applied works have so far been limited to few domains including Reinforcement Learning (Du et al., 2020), reasoning in visual question answering (Kim & Lee, 2019), and physics-inspired system identification (Lutter et al., 2019; Greydanus et al., 2019; Finzi et al., 2020).

heading: 3 NEURAL ORDINARY DIFFERENTIAL EQUATIONS (NEURAL ODE)
text: Neural ODE proposed by Chen et al. (2018) provides a framework for modelling an inherently continuous-time process by means of its time derivative. Given a continuous-time process s(t) ∈ Rd over a defined time interval [t0, t1], we can approximate its underlying true dynamics by learning the parameters of a neural networkFΘ(·) acting as a proxy for the same
ṡ(t) = FΘ(t, s) ∈ Rd. (1) The time derivative of a vector-valued function, i.e. ṡ(t) defines a vector field over the state space Rd which guides the evolution of the process over time. Given the initial state s0 := s(t = t0) and the learned set of parameters Θ∗, the process can be recovered by solving an Initial Value Problem (IVP) using any Differential Equation solver (e.g. RK4 method)
s(t) = s0 + ∫ t t0 FΘ∗(t, s) · dt ∀t ∈ [t0, t1]
Given a set of time-varying functions, one can learn dynamics FΘ capturing high-level concepts. The learning algorithm provided by Chen et al. (2018) makes it practical to train such models with constant memory cost (w.r.t time horizon) both in forward and backward pass.

heading: 4 FRAMEWORK OVERVIEW
text: We borrow the notation of Section 3 and denote chirographic structures as continuous functions of time s(t) defined within [t0, t1]. The exact form of s(t) must include the running 2D coordinates over time and may include other properties like pen-state (whether ink is visible at time t) etc. In principle, one could directly model ṡ(t) with a Neural ODE as in Eq. 1 (see Fig. 2). However, such a model would possess low representational power resulting in underfitting, and furthermore would be incapable of representing self-overlaps naturally exhibited by chirographic data.

heading: 4.1 NEURAL ODE AS DECODER
text: Augmented ODE. We first augment (Dupont et al., 2019) the original state s(t) with the underlying hidden process h(t) in the same time interval. Unlike discrete-time analogues like recurrent networks which model discrete transition of the hidden state followed by an observation model p(s|h), we model the “co-evolution” of the hidden process along with data using a dynamical system defined over the joint space a(t) := [s(t) h(t)]T . Augmented ODEs (Dupont et al., 2019) improve representational power by lifting the vanilla ODE of Eq. 1 to a higher dimension, allowing the data
trajectory to self-overlap in time. In addition, an augmented latent state h(t) provides a principled way to control the trajectory by simply solving the dynamics with a different initial condition h0.
Higher-Order Dynamics. To further increase representational power, we capture the secondorder derivative ä(t) with a parameterized neural network FΘ(·) like Yildiz et al. (2019). This formulation however, leads to the requirement of an extra initial condition ȧ0 := ȧ(t = t0)[
a(t) ȧ(t)
] = [ a0 ȧ0 ] + ∫ t t0 [ ȧ(t) FΘ(·) ] dt (2)
Traditionally, the quantities a(t) and ȧ(t) are termed as “position” and “velocity” respectively of the dynamical system represented by ä(t) = FΘ(·). Inspired by SketchRNN (Ha & Eck, 2018), we simplify the model by dropping the position component a(t) as a dependency to the dynamics function, leading to a dynamics of ä(t) = FΘ(ȧ(t), t). Conditioning. Our model so far is deterministic given a particular choice of initial condition. In order to generate different samples, we need to condition the decoding on a latent vector. Thus, we introduce the following architectural changes:
1. We define a global latent vector z for each sample and compute the hidden part of initial conditions by projecting it through Ldec, as H0 := [h0 ḣ0]T = Ldec(z). Without loss of generality, we fix the visible part of the initial state, i.e. s0 and ṡ0 to be constants.
2. We further include z directly as a fixed parameter to the dynamics function. This is a specific form of data-controlled dynamics (Massaroli et al., 2020). Please see Appendix B for implementation details.
The final second order dynamical system represented by our decoder is therefore ä(t) = FΘ ( ȧ(t), t, z ) (3)
Computing the forward pass (Eq. 2) requires the knowledge of the exact value of z. The latent code can be computed by employing a parameterized encoder. Similar to discrete-time analogues, we propose to employ a non-causal model as an encoder.

heading: 4.2 NEURAL CDE AS ENCODER
text: We propose to use Neural Controlled Differential Equations or Neural CDEs (Kidger et al., 2020) to map a given data sample s(i)(t) to its latent code z.
Neural CDEs are dynamical systems where the hidden state g(t) evolves under the control of a time-varying process. Given a real data sample from the dataset D = {s(i)(t)}Ni=1, an initial state g0 := g(t = t0) is transformed by running an ODE solver on a surrogate ODE dynamics defined as
ġ(t) = GΦ(t,g) ṡ(i)(t)
where GΦ is a parameterized neural network that is analogues to a Bidirectional-RNN encoder in discrete-time. The latent code for the data sequence s(i)(t) is a mapping Lenc of the state that fully encodes the incoming data, i.e. g1 := g(t = t1), which is given by
z = Lenc(g1), where g1 = g0 + ∫ t1 t0 GΦ(t,g) ṡ(i)(t), (4)
where GΦ being a function of time facilitates the usage of non-uniformly sampled data at inference. As suggested by Kidger et al. (2020), the velocities of data trajectory ṡ(i)(t) can be computed with any non-causal interpolation (e.g. Natural Spline curve) of the original discrete data. Alternatively, with some compromise to accuracy, crude estimates can also be computed from discrete data using forward/backward/central difference operator. Refer to Fig. 3 for an overview of the architecture.

heading: 4.3 LOSS FUNCTION & TRAINING
text: Given a ground truth trajectory s(i)(t), and the reconstructed trajectory â(i)(t) (or implied ŝ(i)(t)) computed by executing the encoder and decoder above, we can simply minimize a regression loss (w.r.t Θ,Φ,Lenc,Ldec) at any granularity depending on availability of data and quality requirement
Lfit ( â(i)(t), s(i)(t) ) = ∫ t1 t0 L ( ŝ(i)(t), s(i)(t) ) dt ≈ ∑ t∈[t0,t1] L ( ŝ(i)(t), s(i)(t) ) (5)
Discussion. Note that the solution trajectory computed by Eq. 2 is entirely deterministic given the latent code. However, the end-to-end continuity of the formulation allows discovering structured latent space by default. The mapping from the input s(i) to latent code z, and z to predicted trajectory ŝ(i)(t) is continuous and smooth as long as the dynamics functions governing the encoder CDE and decoder ODE are lipschitz continuous. Consequently, an infinitesimally small perturbation in latent space leads to an infinitesimally small change in the predicted trajectory. This property is important because it enforces structure in the latent space, enables latent space interpolations, and can be exploited to sample new data – as we will show in Section 5.
Our model can also be instantiated as a stochastic generative model (Kingma & Welling, 2014; Ha & Eck, 2018) by assuming a standard isotropic Gaussian prior on the latent code and parameters of the dynamics. Please refer to Appendix A for the exact factorization of the joint distribution and the resulting ELBO objective Lprob. However, we focus on the deterministic variant in the main paper in order to investigate the unique properties it exhibits, as mentioned in Section 1. This also has the benefit of leading to relatively simple learning dynamics due to the absence of VAE-style KL-divergence loss (Kingma & Welling, 2014), which is known to be notoriously challenging to optimize and require specialized engineering tricks (Higgins et al., 2017; Bowman et al., 2016).

heading: 4.4 DATA FORMAT
text: Our model does not make any assumption on the exact form of s(t). However, we propose two practical data format which might be useful for different applications and lead to different trade-offs between computational cost and representation power.
Full-sequence format: s(t) is comprised of two components, the position of the pen x(t) and the state of the pen y(t) ∈ {0, 1} which denotes whether the ink should be rendered or not. Following Section 4.1, we fix x0 = 0 and y0 = 0. Please note that y(t) is different to the traditional “penchange” state popularized by Ha & Eck (2018) which denotes whether the pen switched its state.
Multi-stroke format: Alternatively, we may represent each sample as a set of its constituent strokes {x1(t),x2(t), · · · }, where x := [x y]T ∈ R2, each of which is represented as continuous time functions. In order to incorporate such data format, a slightly modified version of the model is required. For CDE based encoder, we encode every stroke xk(t) individually using Eq. 4 with initial state set to an updated version of the last state from previous stroke with a simple transformation. A similar approach is followed for the decoder ODE with the end of sample decided using a binary flag predicted from last state of a stroke. Please refer to Appendix C for detailed description.

heading: 4.5 A FLEXIBLE FAMILY OF ODE DYNAMICS
text: A vital component of the model with regard to capturing chirographic data properly is the family of parameterized dynamics FΘ and GΦ. So far, the dynamics functions have been chosen to be MLPs (Chen et al., 2018; Massaroli et al., 2020) with standard activation functions like ReLU(·), Tanh(·) and Sigmoid(·), even for time-series data (Kidger et al., 2020; Yildiz et al., 2019). This class of dynamics functions are restricted in terms of the complexity of trajectories they can model and not flexible enough to capture high frequency temporal changes (e.g. multiple overlaps in short time, sharp edges etc.) in the data trajectories. Inspired by SIREN (Sitzmann et al., 2020), which introduced a way to increase spatial frequency content, we propose to use periodic activation functions in order to increase the temporal frequency content of solution trajectories. We use sinusoidal functions, i.e. SIN(ω · x) or COS(ω · x) as pointwise non-linearities. ω is a parameter of the activation function and controls the extent of flexibility of the dynamics function. ω can be treated as hyperparameter or can be learned individually for each layer. Fig. 4 (left) visualizes the dynamics functions induced by MLPs with different activation functions as well as the effect of ω on periodic activation. By controlling ω, it is possible to reduce the frequency content of the solution trajectories under the model, leading to an “abstraction effect” (see Sec. 5.5).

heading: 4.6 CONTINUOUS DATA AUGMENTATION
text: Data augmentation is performed for increasing the effective number of samples by creating synthetic ones from real data. It aims to create a data cloud around the real samples, helping the model discover local manifolds. Discrete time chirographic models are trained primarily by augmenting sequential data with independent Gaussian noise on its constituent points (Ha & Eck, 2018). We propose a new data augmentation strategy specifically fit for continuous time sequences. We use Perlin Noise (Perlin, 1985; 2002), a popular continuous noise algorithm heavily used in Procedural Generation (Lagae et al., 2010), to sample continuous function p(t) ∈ R2 in the same time interval as a given s(t). The noise function is inherently continuous but can be sampled at any necessary granularity. The noise function is then centered (by subtracting its center of gravity) and added to coordinates of the original sample. Refer to Fig. 4 (right) for visual examples. The simplest way to sample an augmented version is by adding σ · (p(t)−pCG) to the constituent strokes of s(t), where
p(t) = [ Perlin2D(s(t) + x + ) Perlin2D(s(t) + y + ) ] and pCG = ∫ t1 t0 p(t)dt
The Perlin2D(·) is the original 2D Perlin Noise function in two dimension, [ x, y] and are coordinate and sample level seeds to simulate pseudo-random behavior. σ is the strength of the noise.

heading: 5 OBSERVATIONS & EXPERIMENTS

heading: 5.1 DATASETS & SETUP
text: We experimented with three chirographic datasets to evaluate our model and compare it to traditional discrete time models. We use a 80%− 20% train-test split for each. VectorMNIST: We collected a small scale dataset of vectorized MNIST digits, i.e. handwritten 0 to 9. We recorded dense points along the trajectory to capture its functional form with least degree of error. A total of 1000 instances of MNIST digits were collected. The dataset is then expanded synthetically by augmenting the real data the with Perlin Noise, as explained in Section 4.6.
QuickDraw:1 Released by Ha & Eck (2018), Quick, Draw! is the largest collection of publicly available free-hand doodling dataset. These doodles are casual representations of familiar concepts drawn within a short amount of time, and are diverse and free-flowing in nature. We use a few categories (face, fish, flower) in order to compare our model against traditional discrete time alternatives.
DiDi Dataset2: Released by Gervais et al. (2020), Digital Ink Diagram dataset is a large collection of synthetically generated flowcharts comprising of different components connected by arrows and lines. It has previously been used for modelling compositional relations by Aksan et al. (2020). We use the “unlabeled” version of DiDi dataset which doesn’t have labels drawn on the diagrams.
Implementation Details To train our framework in practice, we require discretized but possibly non-uniformly sampled ground-truth data along time. For the full-sequence format, each sample in the dataset is represented by an N length sequence each including pen-states. For multi-stroke, each data sample may consists of multiple strokes each represented as a discrete coordinate sequence. For computational and representational convenience, we re-scale the time-range for both full-sequence and every stroke in multi-stroke format to [0, T ] with T being a hyperparameter. For time-sensitive applications, time can be left as it is in the raw data and can be regressed from the latent vector (full-sequence format) or initial decoder state (multi-stroke format). Please refer to Appendix D for a full description of the data structures, resampling and batching mechanism used in practice.
Computing the latent code using CDE based encoder requires a functional form s(t) at training and inference. We use Natural Spline as suggested by Kidger et al. (2020) to construct a spline function from discrete data in order to compute the time-derivative in Eq. 4. For both encoder and decoder, the forward passes involve computing the ODE solution using a black-box ODE solver. Instead of a more accurate adaptive solver, we use RK4 (The classic Runge-Kutta method) as it strikes a perfect balance between computation efficiency and solution accuracy.
The pointwise regression loss L in Eq. 5 is chosen to be Huber Loss or Smooth L1 loss (Girshick, 2015) and the summation is computed for a fixed sequence of uniform time points with granularity Gseq for full-sequence format and Gstroke for multi-stroke format. The granularity does not affect memory consumption as both forward and backward pass of Neural ODE models are constantmemory wrt to the number of points to be evaluated. Due to complex nature of data, Quick, Draw! and DiDi are trained using multi-stroke model with Gstroke = 30; otherwise Gseq = 75, Gstroke = 25. The dimensionality of the hidden state in CDE/ODE is denoted d and the latent dimension as l. For experimentation, we choose d = 64, l = 48 for VectorMNIST, d = 96, l = 84 for DiDi and d = 144, l = 96 for Quick, Draw!. Dynamics networks (for both CDE and ODE) are MLPs with few (2 for full-sequence, 1 for multi-stroke) hidden layers of size d1.2de. For a fair comparison with discrete time models, we use an LSTM-LSTM Seq2Seq model with same hidden dimension. We empirically found the best value of the upper limit of time range to be T = 5. All models are trained with AdamW (Loshchilov & Hutter, 2019) optimizer, learning rate of 1×10−4 for RNN and 3 × 10−3 for CDE-ODE, annealed using cosine scheduler (100 epoch period) with decreasing (by factor of 4) amplitude. We also used 10−2 as weight decay (L2 penalty) regularizer on the weight of the dynamics networks and gradients are clipped at a magnitude of 0.01.
5.2 RECONSTRUCTION & GENERATION TASKS
We first evaluate our framework for the traditional reconstruction task and compare with discrete time RNN model. Reconstruction performance is an indicator of a model’s ability to faithfully capture an input by means of a compact latent code. We compare our deterministic SketchODE model with an RNN-RNN Seq2Seq model trained for deterministic reconstruction. We also train a recognition CNN on the train data and apply it to the rasterized reconstruction of test data as a semantic measure of reconstruction fidelity. We com-
pute test loss and recognition accuracy for both model on all datasets. The results (in Table. 1) show that SketchODE performs comparably to standard RNN-RNN Seq2Seq models. While recon-
1https://github.com/googlecreativelab/quickdraw-dataset 2https://github.com/google-research/google-research/tree/master/didi dataset
struction loss is slightly worse, SketchODE retains recognizable visual semantics. This confirms that continuous time models have a strong inductive bias towards semantic content rather than exact point-wise reconstruction. We also compare with two related alternatives: BézierSketch (Das et al., 2020) and CoSE (Aksan et al., 2020) trained deterministically.
We also validated our design choices of periodic activations and continuous noise augmentation for reconstruction task with an ablation study shown in Figure. 5 (middle).
The generative model trained with an extra Lprob (as described in Section 4.3) as loss can generate plausible data even unconditionally (Refer to Appendix A). However, a special property of continuous time Seq2Seq models like ours is conditional sampling from a deterministically trained network. Since the mapping from latent code to output is defined by an ODE solution trajectory, infinitesimal perturbation of latent codes leads to similar perturbation in the data, thus preserving semantic structure. For the discrete counterpart i.e. RNN Seq2Seq models, the sampling-based autoregressive nature of inference makes the latent to output map discontinuous. For CoSE (Aksan et al., 2020), even if the the individual strokes remain continuous, their geometry and relational structures (autoregressive) break down. We deterministically encode input data into latent code z and sample from N (z, σI). Figure 5 (left) shows conditional sampling from both type of networks with σ = 0.05. The RNN fails to retain both smoothness and semantics, but samples from our SketchODE are diverse, visually pleasing, and meaningful.

heading: 5.3 DATA EFFICIENCY
text: Temporally continuous models are data efficient for chirographic representation due to their strong inductive bias towards continuous data. The implicit distribution defined by our SketchODE model has a support only over all possible continuous trajectories. Discrete time sequence models on the other hand, define a distribution over all possible trajectories and hence are required to “learn” the bias (over continuous trajectories) from data itself. To illustrate this we train both RNN-RNN and our SketchODE model on varying amounts of data. Dimensionality of state/hidden vectors are kept to be same for fair comparison. Fig. 5 (right) shows the test reconstruction error as a function of training data. We can see that SketchODE has a particular advantage in the low data regime.

heading: 5.4 INTERPOLATION & ANIMATION
text: Continuing the discussion about the strong inductive bias toward continuous data, SketchODE cannot generate temporal discontinuity for any latent vector z. This leads to the property that smooth interpolation in latent codes leads to smooth interpolation in the data. This is a noteworthy properly for a deterministic autoencoder like SketchODE as it is not true for traditional RNN-RNN models. To illustrate this we train SketchODE and extract latent codes z0 and z1 for two data samples s(1) and s(2). We interpolate between latent codes and produce zinterp = z0∗α+z1∗(1−α) and decode zinterp for several values of α ∈ [0, 1]. Fig. 6 (left, right) illustrate interpolation in VectorMNIST, QuickDraw and DiDi datasets. More examples are shown in Appendix E.
One-shot interpolation between unseen classes Existing latent models like (Ha & Eck, 2018) need to be trained on substantial amounts of data in order to learn the manifold required for latentspace exploration. A unique capability of SketchODE is the ability to perform interpolation after
1-shot learning of new categories. This is due to the strong inductive bias of continuity that must be learned from scratch in discrete-time analogues. We demonstrate this by creating two extra handwritten samples “A” and “B”, and fine-tune our trained VectorMNIST model for approx 1k iterations. We are then able to interpolate between novel letters as shown in Fig. 6(middle).

heading: 5.5 ABSTRACTION EFFECT
text: Strong inductive bias towards temporal continuity enables another capability which we term the “abstraction effect”. In the context of sketch, “abstraction” (Berger et al., 2013; Muhammad et al., 2019) refers to removing details from the data while keeping as much semantic content as possible. Our proposed continuous SketchODE model is a perfect fit for creating such abstraction. As discussed before, SketchODE can use almost the entirety of its capacity for modelling the semantic content since temporal continuity comes for free. We observed that by reducing the capacity of the decoder, we can systematically control the mount of detail that can be captured by the model. Due to our specific choice of dynamics family, an easy way to do so is by controlling the frequency of periodic activation functions. We use SIN(·) as activation due to the fact that ω → 0⇒ SIN(ω · x)→ x (i.e. linear activation) and hence the model looses most of its capacity. Figure 7 shows that as we reduce activation function frequency, high frequency content is removed from the data in the reconstruction.

heading: 6 CONCLUSIONS
text: In this paper, we proposed a fundamentally new paradigm for representing chirographic drawing data like handwriting, diagrams, and sketches. The SketchODE framework and its unique functional data representation posses several qualities that its discrete-time analogues do not: generation of inherently scalable arbitrary-resolution images, data efficient learning due to a built-in inductive bias towards continuity, ability to conditionally sample and perform 1-shot interpolation with a deterministically trained model. One drawback however is the computational complexity of training which require computing higher order derivatives in both forward and backward pass. Despite computational challenges, this work may provide a conceptual primitive for modelling more complex but constrained structures (e.g. fonts, icon and vector arts). More generally, this is the first continuoustime Seq2Seq model, which may find application in diverse domains beyond chirography.

heading: B DATA-CONTROLLED DYNAMICS
text: A dynamical system can be controlled by a parameter that do not “evolve” over time but can be differentiated with. Such design is loosely inspired by SketchRNN (Ha & Eck, 2018) which concatenates decoder inputs with the global latent vector in order to prevent the recurrence from forgetting the initial state. With the dynamics function FΘ(ȧ, t, z), we can easily achieve this in practice by solving the following dynamical system
ä(t) = F̂Θ ( [[ȧ t]T
z
] ) = [ FΘ(ȧ, t, z)
0 ] The fixed zero vector as second derivative of z prevents it from co-evolving along with ȧ. Since z is now a part of the state, Adjoint Backpropagation (Chen et al., 2018) naturally allows easy computation of the partial derivative.

heading: C MULTI-STROKE FORMAT
text: Unlike full-sequence format, we use the architecture defined in the main paper for processing one single stroke at a time while sharing the parameters across different strokes. Refer to Fig. 10 for a concise diagram.
For CDE based encoder, a sequence of strokes {x1(t),x2(t), · · · } is processed individually with a CDE with initial state gk0 := g k(t = t0) and final state gk1 := g k(t = t0). The time range for each stroke, if not available in dataset, can be fixed as a hyperparameter. Initial state for the kth stroke can be computed by projecting the last state of (k − 1)th stroke with parametric transform U (e.g. MLP)
gk0 = U(gk−11 ), where g k−1 1 = g k−1 0 + ∫ t1 t0 GΦ(t,g) ẋk−1(t) · dt
The initial state of the very first stroke is always fixed to a constant, e.g. g00 = 0.
For ODE based decoder, a similar strategy is followed with the exception that the initial state of the first stroke is set to A0 where ṡ0 = s0 = 0 and H0 = Lenc(z). The initial state of the kth stroke is produced by transforming the last state of (k − 1)th stroke with another parametric transform V (e.g. MLP)
Hk0 = V(Ak−11 ), where
Ak−11 = A k−1 0 + ∫ t1 t0 [ ȧ(t) FΘ ] dt
Please note that for every stroke, the initial position and velocity is fixed to zero. Such a strategy leads to the ODE dynamics learning shared stroke model. It follows earlier works like Das et al. (2020; 2021); Aksan et al. (2020) and require a starting position vk for every stroke to be predicted separately, which can be regressed during training. Also, an end-of-sample flag is necessary for terminating the inference which can be achieved by defining a random variable rk ∈ {0, 1} as whether the kth stroke is the last stroke and predicting the probability of it being one
p(rk = 1|Ak1 ; θ) = Sigmoid(Rθ(Ak1))

heading: D PRACTICAL DATA STRUCTURES & MINI-BATCHING
text: Samples in full-sequence data format is represented by a length N sequence [(tj , s(t = tj))]Nj=1 where tj ∈ [0, T ] are non-uniform time-points. For multi-stroke format, each data sample may consists of multiple strokes each represented as a discrete coordinate sequence. The end coordinate of the previous stroke and starting coordinate of the following stroke are connected via straight line and y(t = tj) is set to 1 for all time points on it; otherwise y(t = tj) = 0. Please refer to Fig. 11 for visual illustration. For more time-sensitive applications, the time-range can be left as it is in the raw data and can be regressed from the latent vector (full-sequence format) or initial decoder state (multi-stroke format). At inference, given a latent code, the decoder ODE can be solved using the predicted time-range.
In practice, every sample in the dataset needs to have a format shown in Fig. 11. For full sequence format, every stroke including the pen-up strokes (i.e. joining line between previous stroke-ending and next stroke-begining) are kept in order. The pen-up stroke is shown in gray and the corresponding y(t) is set to one. The multi-stroke format do not require any y(t). All samples are spatially scaled down inside the unit circle, i.e. ‖x(t)‖2 ≤ 1 ∀t ∈ [0, T ]. For datasets with high temporal sampling rate (e.g. DiDi and Quick, Draw!), we reduce it by Ramer–Douglas–Peucker algorithm (Douglas & Peucker, 1973) as a preprocessing step.
Efficient training of Neural ODE/CDE models further require mini-batching, which is different than traditional models. For samples with different number of time-points, we create a common sequence of time-points by union-ing individual time-point sequences. An ODE solution trajectory is then evaluated at the time points in the union and masking is done to discard irrelevant time points for each sample. For even more efficiency, we can re-sample each data in a mini-batch at a fixed sequence of uniform time-points. This is achieved by linear interpolation of points along time. Please refer to Fig. 11 for visual illustration.
E VISUAL SAMPLES

Review summaries:
The paper describes a way of representing online handwriting data, typically seen as a discrete sequence, in a continuous space, using neural ODEs. The learned representation allows sampling from and interpolation in the latent space. The results in the paper are compared (mostly qualitatively) to autoencoder-learned representations based on representing data as discrete sequences, Bezier curves, and implicit surface representation from differentable geometrty.

This paper proposes a new model, called SketchODE, for learning representations of sketches using neural ODEs. Specifically, the authors parameterize hand drawn strokes as solutions of ODEs and build an auto encoder like framework for learning the vector fields of these ODEs from data.   The decoder is modelled with a second order neural ODE acting on an augmented state (which allows for self overlapping trajectories which is necessary for handwritten data) that includes the stroke itself as well as an underlying hidden state. The decoder is made conditional by passing a latent vector to the model, which is mapped both to the initial state of the ODE as well as the dynamics function of the ODE. Varying this latent vector and solving the ODE then gives rise to different handwritten patterns.  The encoder is parameterized by a neural CDE which takes as input some ground truth trajectory and returns a new state which is mapped to the latent vector. The model is then trained end to end using a reconstruction loss between the ground truth and reconstructed trajectories.  In addition, the authors show that two additional tricks can further improve performance: using sine activation functions in the MLPs parameterizing the vector fields as well as using perlin noise to create (continuous) augmentations of the data.  The authors then discuss and demonstrate various compelling properties of their model. This includes the existence of a latent space (which allows for interpolation of handwritten data) as well as one shot learning capabilities for new classes of data.  The main contributions of the paper in my eyes are then: - Introducing an interesting continuous neural model of handwritten data - Demonstrating compelling properties of this continuous model over traditional discrete models - Introducing the VectorMNIST dataset, which I believe could be useful for other researchers - Interesting experiments on various handwritten datasets