SketchODE: Learning neural sketch representation in continuous time

1 INTRODUCTION Drawing-based communications such as sketches , writing , and diagrams come naturally to humans and have been used in some form since ancient times . Modeling such data is becoming an increasingly important and topical challenge area for machine learning systems aiming to interpret and simulate human creative expression . Such chirographic structures are challenging to interpret and generate due to their complex and unstructured nature . However recent progress has been strong , thanks to the advances in learning latent representations for sequences ( Bowman et al. , 2016 ; Graves , 2013 ; Srivastava et al. , 2015 ) . In particular , with the advent of variational sequence-to-sequence generative models ( Srivastava et al. , 2015 ; Bowman et al. , 2016 ) and the collection of largescale datasets ( Gervais et al. , 2020 ; Ha & Eck , 2018 ; Ge et al. , 2021 ) , we have seen a surge of advancements ( Ha & Eck , 2018 ; Aksan et al. , 2020 ) in this direction . Neverthe- less , a key missing link is the fact that drawing is intrinsically continuous in time , as is the resulting drawn artefact , when considered as a sequence rather than a raster image . The dominant approach to model free-hand sketches , popularized by Ha & Eck ( 2018 ) , has been to use a discrete-step recurrent network for capturing the latent process p ( h [ n ] |h [ n− 1 ] ) with discrete step n , and a complex observation model p ( s [ n ] | h [ n ] ) to explain local structure . The large body of approaches built upon this have a critical flaw : they ignore the very nature of the data , i.e . drawn structures are inherently continuous in time ( Refer to Fig . 1 ) . While a few attempts have been made to use continuous-time constructs like Bézier curves ( Das et al. , 2020 ; 2021 ) , their inconvenient mathematical form affect the representational power as well as training dynamics . The reason for such an absence of continuous-time models is the lack of fundamental tools for handling continuous time data . Lately however , the introduction of Neural Ordinary Differential Equations by Chen et al . ( 2018 ) followed by numerous extensions ( Dupont et al. , 2019 ; Yildiz et al. , 2019 ; Kidger et al. , 2020 ) have opened the possibility of building powerful models that can natively represent continuous time data such as handwriting , sketch etc . In this paper , we represent chirographic drawing data like handwriting , sketches etc . as continuoustime function s ( t ) and model the underlying latent process also as continuous-time latent function h ( t ) . Specifically , we propose a framework for capturing the derivative ( potentially higher order ) of the latent process dKh ( t ) dtK with parameterized neural networks . At inference , a solution trajectory ŝ ( t ) is computed from a given Initial Value Problem ( IVP ) that includes a learned ODE dynamics and an initial hidden state h0 , h ( t = 0 ) . An obvious advantage of the functional representation is its arbitrary spatial resolution , i.e . we can retrieve a structure at any spatial resolution by sampling the function at appropriate rate . Moreover , with a functional representation , we can systematically upgrade the representation along time with additional properties ( e.g . line thickness , color etc ) . So far , parameterized ODE dynamics models have largely been treated as a “ replacement for ResNets ” ( Massaroli et al. , 2020 ) where the intermediate states are of little importance . While the capability of ODE models are beginning to be tested in time-series data ( Kidger et al. , 2020 ; Morrill et al. , 2021 ) , they still remain largely unexplored . Following the introduction of Neural ODEs by Chen et al . ( 2018 ) , methods have been proposed in order to regularize the solution trajectory to be as simple as possible ( Kelly et al. , 2020 ; Finlay et al. , 2020 ) . However , time-series data such as chirography are entirely the opposite . The dynamics model , and consequently the solution trajectory should be as flexible as possible in order to learn the high degree of local and global variations with time often exhibited in such data . We increase the flexibility of Neural ODE models by introducing a new class of parameterized dynamics functions . Our experiments show that this is crucial in order to model chirographic data with satisfactory fidelity . We also introduce a new data augmentation method particularly geared towards ODE models and continuous-time sequences . Finally , we propose a deterministic autoencoder with a global latent variable z that exhibits some generative model properties by virtue of inherent continuity . Our final model , SketchODE , is similar to SketchRNN ( Ha & Eck , 2018 ) in terms of high-level design but lifts it to the realm of continuoustime . More generally , our model is the first continuous time Seq2Seq architecture . We explore some of the noteworthy features that differentiate SketchODE from discrete-time Seq2Seq analogoues . An inductive bias for continuity : Discrete-time sequence models ( Ha & Eck , 2018 ) need to use their capacity to model both global structure and temporal continuity in the data . ODE models on the other hand , have a strong intrinsic architectural bias towards temporal continuity . Sequences generated using a well-behaved ODE dynamics function are guaranteed to be continuous . Since they need not learn the continuity bias from scratch , ODE models are more data-efficient and are able to use majority of their capacity for modelling higher level structures . We even demonstrate that our SketchODE supports meaningful 1-shot learning when fine-tuned . Deterministic Autoencoding with Structured Latent Space : A surprising property of Seq2Seq ODE models is the ability to perform latent space interpolation and sampling without the necessity of imposing a prior distribution p ( z ) on the variational posterior qφ ( z|s ) . This property is also a consequence of the latent to output mapping being continuous . 2 RELATED WORK . With the advent of recurrent neural networks , sequential time-series models have seen significant adaptation in a variety of applications . Video ( Srivastava et al. , 2015 ) and natural language ( Bowman et al. , 2016 ) were two early applications to have seen significant success . A different but important time-series modality is chirographic data like handwriting ( Graves , 2013 ) and sketches ( Ha & Eck , 2018 ) which gained traction only recently . Subsequent developments also address data modalities that are not free-flowing , like Fonts ( Lopes et al. , 2019 ) and Icons ( Carlier et al. , 2020 ) which require slightly different models . Mirroring developments in natural language processing ( Vaswani et al. , 2017 ) , Transformers are now replacing ( Carlier et al. , 2020 ) recurrent networks in such models . Following the seminal work of Ha & Eck ( 2018 ) , the primary representation for chirographic data has been discrete sequence of waypoints or tokens from specialized Domain Specific Languages like SVG ( Scalable Vector Graphics ) , with a minority continuing to make use of standard raster graphic architectures ( Ge et al. , 2021 ) . Overall , few studies have attempted to develop better representations using specialized functional forms , with Aksan et al . ( 2020 ) learning stroke embeddings and Das et al . ( 2020 ; 2021 ) directly modeling strokes as parametric Bézier curves . However , to the best of our knowledge , modelling chirographic data as generic functions have not yet been tried so far . The Neural ODE ( Chen et al. , 2018 ) framework provides a credible tool for natively modelling continuous time functions using dynamical systems . Since their inception , Neural ODEs have sparked a wide range of discussions and developments ( Massaroli et al. , 2020 ) , mostly in the theoretical domain . Neural ODEs have been extended to work with latent states ( Dupont et al. , 2019 ) , to define generative models ( Song et al. , 2021 ) , and have external data control ( Kidger et al. , 2020 ) . While practical training of ODE models remain a challenge , significant amounts of work have been dedicated towards developing engineering techniques ( Finlay et al. , 2020 ; Grathwohl et al. , 2019 ; Kelly et al. , 2020 ; Poli et al. , 2020 ) for faster training convergence and better regularization . Despite flourishing theoretical interests , due to high training complexity , applied works have so far been limited to few domains including Reinforcement Learning ( Du et al. , 2020 ) , reasoning in visual question answering ( Kim & Lee , 2019 ) , and physics-inspired system identification ( Lutter et al. , 2019 ; Greydanus et al. , 2019 ; Finzi et al. , 2020 ) . 3 NEURAL ORDINARY DIFFERENTIAL EQUATIONS ( NEURAL ODE ) . Neural ODE proposed by Chen et al . ( 2018 ) provides a framework for modelling an inherently continuous-time process by means of its time derivative . Given a continuous-time process s ( t ) ∈ Rd over a defined time interval [ t0 , t1 ] , we can approximate its underlying true dynamics by learning the parameters of a neural networkFΘ ( · ) acting as a proxy for the same ṡ ( t ) = FΘ ( t , s ) ∈ Rd . ( 1 ) The time derivative of a vector-valued function , i.e . ṡ ( t ) defines a vector field over the state space Rd which guides the evolution of the process over time . Given the initial state s0 : = s ( t = t0 ) and the learned set of parameters Θ∗ , the process can be recovered by solving an Initial Value Problem ( IVP ) using any Differential Equation solver ( e.g . RK4 method ) s ( t ) = s0 + ∫ t t0 FΘ∗ ( t , s ) · dt ∀t ∈ [ t0 , t1 ] Given a set of time-varying functions , one can learn dynamics FΘ capturing high-level concepts . The learning algorithm provided by Chen et al . ( 2018 ) makes it practical to train such models with constant memory cost ( w.r.t time horizon ) both in forward and backward pass . 4 FRAMEWORK OVERVIEW . We borrow the notation of Section 3 and denote chirographic structures as continuous functions of time s ( t ) defined within [ t0 , t1 ] . The exact form of s ( t ) must include the running 2D coordinates over time and may include other properties like pen-state ( whether ink is visible at time t ) etc . In principle , one could directly model ṡ ( t ) with a Neural ODE as in Eq . 1 ( see Fig . 2 ) . However , such a model would possess low representational power resulting in underfitting , and furthermore would be incapable of representing self-overlaps naturally exhibited by chirographic data . 4.1 NEURAL ODE AS DECODER . Augmented ODE . We first augment ( Dupont et al. , 2019 ) the original state s ( t ) with the underlying hidden process h ( t ) in the same time interval . Unlike discrete-time analogues like recurrent networks which model discrete transition of the hidden state followed by an observation model p ( s|h ) , we model the “ co-evolution ” of the hidden process along with data using a dynamical system defined over the joint space a ( t ) : = [ s ( t ) h ( t ) ] T . Augmented ODEs ( Dupont et al. , 2019 ) improve representational power by lifting the vanilla ODE of Eq . 1 to a higher dimension , allowing the data trajectory to self-overlap in time . In addition , an augmented latent state h ( t ) provides a principled way to control the trajectory by simply solving the dynamics with a different initial condition h0 . Higher-Order Dynamics . To further increase representational power , we capture the secondorder derivative ä ( t ) with a parameterized neural network FΘ ( · ) like Yildiz et al . ( 2019 ) . This formulation however , leads to the requirement of an extra initial condition ȧ0 : = ȧ ( t = t0 ) [ a ( t ) ȧ ( t ) ] = [ a0 ȧ0 ] + ∫ t t0 [ ȧ ( t ) FΘ ( · ) ] dt ( 2 ) Traditionally , the quantities a ( t ) and ȧ ( t ) are termed as “ position ” and “ velocity ” respectively of the dynamical system represented by ä ( t ) = FΘ ( · ) . Inspired by SketchRNN ( Ha & Eck , 2018 ) , we simplify the model by dropping the position component a ( t ) as a dependency to the dynamics function , leading to a dynamics of ä ( t ) = FΘ ( ȧ ( t ) , t ) . Conditioning . Our model so far is deterministic given a particular choice of initial condition . In order to generate different samples , we need to condition the decoding on a latent vector . Thus , we introduce the following architectural changes : 1 . We define a global latent vector z for each sample and compute the hidden part of initial conditions by projecting it through Ldec , as H0 : = [ h0 ḣ0 ] T = Ldec ( z ) . Without loss of generality , we fix the visible part of the initial state , i.e . s0 and ṡ0 to be constants . 2 . We further include z directly as a fixed parameter to the dynamics function . This is a specific form of data-controlled dynamics ( Massaroli et al. , 2020 ) . Please see Appendix B for implementation details . The final second order dynamical system represented by our decoder is therefore ä ( t ) = FΘ ( ȧ ( t ) , t , z ) ( 3 ) Computing the forward pass ( Eq . 2 ) requires the knowledge of the exact value of z . The latent code can be computed by employing a parameterized encoder . Similar to discrete-time analogues , we propose to employ a non-causal model as an encoder .

Summary: This paper proposes a new model, called SketchODE, for learning representations of sketches using neural ODEs. Specifically, the authors parameterize hand drawn strokes as solutions of ODEs and build an auto encoder like framework for learning the vector fields of these ODEs from data.   The decoder is modelled with a second order neural ODE acting on an augmented state (which allows for self overlapping trajectories which is necessary for handwritten data) that includes the stroke itself as well as an underlying hidden state. The decoder is made conditional by passing a latent vector to the model, which is mapped both to the initial state of the ODE as well as the dynamics function of the ODE. Varying this latent vector and solving the ODE then gives rise to different handwritten patterns.  The encoder is parameterized by a neural CDE which takes as input some ground truth trajectory and returns a new state which is mapped to the latent vector. The model is then trained end to end using a reconstruction loss between the ground truth and reconstructed trajectories.  In addition, the authors show that two additional tricks can further improve performance: using sine activation functions in the MLPs parameterizing the vector fields as well as using perlin noise to create (continuous) augmentations of the data.  The authors then discuss and demonstrate various compelling properties of their model. This includes the existence of a latent space (which allows for interpolation of handwritten data) as well as one shot learning capabilities for new classes of data.  The main contributions of the paper in my eyes are then: - Introducing an interesting continuous neural model of handwritten data - Demonstrating compelling properties of this continuous model over traditional discrete models - Introducing the VectorMNIST dataset, which I believe could be useful for other researchers - Interesting experiments on various handwritten datasets