SketchODE: Learning neural sketch representation in continuous time

1 INTRODUCTION Drawing-based communications such as sketches , writing , and diagrams come naturally to humans and have been used in some form since ancient times . Modeling such data is becoming an increasingly important and topical challenge area for machine learning systems aiming to interpret and simulate human creative expression . Such chirographic structures are challenging to interpret and generate due to their complex and unstructured nature . However recent progress has been strong , thanks to the advances in learning latent representations for sequences ( Bowman et al. , 2016 ; Graves , 2013 ; Srivastava et al. , 2015 ) . In particular , with the advent of variational sequence-to-sequence generative models ( Srivastava et al. , 2015 ; Bowman et al. , 2016 ) and the collection of largescale datasets ( Gervais et al. , 2020 ; Ha & Eck , 2018 ; Ge et al. , 2021 ) , we have seen a surge of advancements ( Ha & Eck , 2018 ; Aksan et al. , 2020 ) in this direction . Neverthe- less , a key missing link is the fact that drawing is intrinsically continuous in time , as is the resulting drawn artefact , when considered as a sequence rather than a raster image . The dominant approach to model free-hand sketches , popularized by Ha & Eck ( 2018 ) , has been to use a discrete-step recurrent network for capturing the latent process p ( h [ n ] |h [ n− 1 ] ) with discrete step n , and a complex observation model p ( s [ n ] | h [ n ] ) to explain local structure . The large body of approaches built upon this have a critical flaw : they ignore the very nature of the data , i.e . drawn structures are inherently continuous in time ( Refer to Fig . 1 ) . While a few attempts have been made to use continuous-time constructs like Bézier curves ( Das et al. , 2020 ; 2021 ) , their inconvenient mathematical form affect the representational power as well as training dynamics . The reason for such an absence of continuous-time models is the lack of fundamental tools for handling continuous time data . Lately however , the introduction of Neural Ordinary Differential Equations by Chen et al . ( 2018 ) followed by numerous extensions ( Dupont et al. , 2019 ; Yildiz et al. , 2019 ; Kidger et al. , 2020 ) have opened the possibility of building powerful models that can natively represent continuous time data such as handwriting , sketch etc . In this paper , we represent chirographic drawing data like handwriting , sketches etc . as continuoustime function s ( t ) and model the underlying latent process also as continuous-time latent function h ( t ) . Specifically , we propose a framework for capturing the derivative ( potentially higher order ) of the latent process dKh ( t ) dtK with parameterized neural networks . At inference , a solution trajectory ŝ ( t ) is computed from a given Initial Value Problem ( IVP ) that includes a learned ODE dynamics and an initial hidden state h0 , h ( t = 0 ) . An obvious advantage of the functional representation is its arbitrary spatial resolution , i.e . we can retrieve a structure at any spatial resolution by sampling the function at appropriate rate . Moreover , with a functional representation , we can systematically upgrade the representation along time with additional properties ( e.g . line thickness , color etc ) . So far , parameterized ODE dynamics models have largely been treated as a “ replacement for ResNets ” ( Massaroli et al. , 2020 ) where the intermediate states are of little importance . While the capability of ODE models are beginning to be tested in time-series data ( Kidger et al. , 2020 ; Morrill et al. , 2021 ) , they still remain largely unexplored . Following the introduction of Neural ODEs by Chen et al . ( 2018 ) , methods have been proposed in order to regularize the solution trajectory to be as simple as possible ( Kelly et al. , 2020 ; Finlay et al. , 2020 ) . However , time-series data such as chirography are entirely the opposite . The dynamics model , and consequently the solution trajectory should be as flexible as possible in order to learn the high degree of local and global variations with time often exhibited in such data . We increase the flexibility of Neural ODE models by introducing a new class of parameterized dynamics functions . Our experiments show that this is crucial in order to model chirographic data with satisfactory fidelity . We also introduce a new data augmentation method particularly geared towards ODE models and continuous-time sequences . Finally , we propose a deterministic autoencoder with a global latent variable z that exhibits some generative model properties by virtue of inherent continuity . Our final model , SketchODE , is similar to SketchRNN ( Ha & Eck , 2018 ) in terms of high-level design but lifts it to the realm of continuoustime . More generally , our model is the first continuous time Seq2Seq architecture . We explore some of the noteworthy features that differentiate SketchODE from discrete-time Seq2Seq analogoues . An inductive bias for continuity : Discrete-time sequence models ( Ha & Eck , 2018 ) need to use their capacity to model both global structure and temporal continuity in the data . ODE models on the other hand , have a strong intrinsic architectural bias towards temporal continuity . Sequences generated using a well-behaved ODE dynamics function are guaranteed to be continuous . Since they need not learn the continuity bias from scratch , ODE models are more data-efficient and are able to use majority of their capacity for modelling higher level structures . We even demonstrate that our SketchODE supports meaningful 1-shot learning when fine-tuned . Deterministic Autoencoding with Structured Latent Space : A surprising property of Seq2Seq ODE models is the ability to perform latent space interpolation and sampling without the necessity of imposing a prior distribution p ( z ) on the variational posterior qφ ( z|s ) . This property is also a consequence of the latent to output mapping being continuous . 2 RELATED WORK . With the advent of recurrent neural networks , sequential time-series models have seen significant adaptation in a variety of applications . Video ( Srivastava et al. , 2015 ) and natural language ( Bowman et al. , 2016 ) were two early applications to have seen significant success . A different but important time-series modality is chirographic data like handwriting ( Graves , 2013 ) and sketches ( Ha & Eck , 2018 ) which gained traction only recently . Subsequent developments also address data modalities that are not free-flowing , like Fonts ( Lopes et al. , 2019 ) and Icons ( Carlier et al. , 2020 ) which require slightly different models . Mirroring developments in natural language processing ( Vaswani et al. , 2017 ) , Transformers are now replacing ( Carlier et al. , 2020 ) recurrent networks in such models . Following the seminal work of Ha & Eck ( 2018 ) , the primary representation for chirographic data has been discrete sequence of waypoints or tokens from specialized Domain Specific Languages like SVG ( Scalable Vector Graphics ) , with a minority continuing to make use of standard raster graphic architectures ( Ge et al. , 2021 ) . Overall , few studies have attempted to develop better representations using specialized functional forms , with Aksan et al . ( 2020 ) learning stroke embeddings and Das et al . ( 2020 ; 2021 ) directly modeling strokes as parametric Bézier curves . However , to the best of our knowledge , modelling chirographic data as generic functions have not yet been tried so far . The Neural ODE ( Chen et al. , 2018 ) framework provides a credible tool for natively modelling continuous time functions using dynamical systems . Since their inception , Neural ODEs have sparked a wide range of discussions and developments ( Massaroli et al. , 2020 ) , mostly in the theoretical domain . Neural ODEs have been extended to work with latent states ( Dupont et al. , 2019 ) , to define generative models ( Song et al. , 2021 ) , and have external data control ( Kidger et al. , 2020 ) . While practical training of ODE models remain a challenge , significant amounts of work have been dedicated towards developing engineering techniques ( Finlay et al. , 2020 ; Grathwohl et al. , 2019 ; Kelly et al. , 2020 ; Poli et al. , 2020 ) for faster training convergence and better regularization . Despite flourishing theoretical interests , due to high training complexity , applied works have so far been limited to few domains including Reinforcement Learning ( Du et al. , 2020 ) , reasoning in visual question answering ( Kim & Lee , 2019 ) , and physics-inspired system identification ( Lutter et al. , 2019 ; Greydanus et al. , 2019 ; Finzi et al. , 2020 ) . 3 NEURAL ORDINARY DIFFERENTIAL EQUATIONS ( NEURAL ODE ) . Neural ODE proposed by Chen et al . ( 2018 ) provides a framework for modelling an inherently continuous-time process by means of its time derivative . Given a continuous-time process s ( t ) ∈ Rd over a defined time interval [ t0 , t1 ] , we can approximate its underlying true dynamics by learning the parameters of a neural networkFΘ ( · ) acting as a proxy for the same ṡ ( t ) = FΘ ( t , s ) ∈ Rd . ( 1 ) The time derivative of a vector-valued function , i.e . ṡ ( t ) defines a vector field over the state space Rd which guides the evolution of the process over time . Given the initial state s0 : = s ( t = t0 ) and the learned set of parameters Θ∗ , the process can be recovered by solving an Initial Value Problem ( IVP ) using any Differential Equation solver ( e.g . RK4 method ) s ( t ) = s0 + ∫ t t0 FΘ∗ ( t , s ) · dt ∀t ∈ [ t0 , t1 ] Given a set of time-varying functions , one can learn dynamics FΘ capturing high-level concepts . The learning algorithm provided by Chen et al . ( 2018 ) makes it practical to train such models with constant memory cost ( w.r.t time horizon ) both in forward and backward pass . 4 FRAMEWORK OVERVIEW . We borrow the notation of Section 3 and denote chirographic structures as continuous functions of time s ( t ) defined within [ t0 , t1 ] . The exact form of s ( t ) must include the running 2D coordinates over time and may include other properties like pen-state ( whether ink is visible at time t ) etc . In principle , one could directly model ṡ ( t ) with a Neural ODE as in Eq . 1 ( see Fig . 2 ) . However , such a model would possess low representational power resulting in underfitting , and furthermore would be incapable of representing self-overlaps naturally exhibited by chirographic data . 4.1 NEURAL ODE AS DECODER . Augmented ODE . We first augment ( Dupont et al. , 2019 ) the original state s ( t ) with the underlying hidden process h ( t ) in the same time interval . Unlike discrete-time analogues like recurrent networks which model discrete transition of the hidden state followed by an observation model p ( s|h ) , we model the “ co-evolution ” of the hidden process along with data using a dynamical system defined over the joint space a ( t ) : = [ s ( t ) h ( t ) ] T . Augmented ODEs ( Dupont et al. , 2019 ) improve representational power by lifting the vanilla ODE of Eq . 1 to a higher dimension , allowing the data trajectory to self-overlap in time . In addition , an augmented latent state h ( t ) provides a principled way to control the trajectory by simply solving the dynamics with a different initial condition h0 . Higher-Order Dynamics . To further increase representational power , we capture the secondorder derivative ä ( t ) with a parameterized neural network FΘ ( · ) like Yildiz et al . ( 2019 ) . This formulation however , leads to the requirement of an extra initial condition ȧ0 : = ȧ ( t = t0 ) [ a ( t ) ȧ ( t ) ] = [ a0 ȧ0 ] + ∫ t t0 [ ȧ ( t ) FΘ ( · ) ] dt ( 2 ) Traditionally , the quantities a ( t ) and ȧ ( t ) are termed as “ position ” and “ velocity ” respectively of the dynamical system represented by ä ( t ) = FΘ ( · ) . Inspired by SketchRNN ( Ha & Eck , 2018 ) , we simplify the model by dropping the position component a ( t ) as a dependency to the dynamics function , leading to a dynamics of ä ( t ) = FΘ ( ȧ ( t ) , t ) . Conditioning . Our model so far is deterministic given a particular choice of initial condition . In order to generate different samples , we need to condition the decoding on a latent vector . Thus , we introduce the following architectural changes : 1 . We define a global latent vector z for each sample and compute the hidden part of initial conditions by projecting it through Ldec , as H0 : = [ h0 ḣ0 ] T = Ldec ( z ) . Without loss of generality , we fix the visible part of the initial state , i.e . s0 and ṡ0 to be constants . 2 . We further include z directly as a fixed parameter to the dynamics function . This is a specific form of data-controlled dynamics ( Massaroli et al. , 2020 ) . Please see Appendix B for implementation details . The final second order dynamical system represented by our decoder is therefore ä ( t ) = FΘ ( ȧ ( t ) , t , z ) ( 3 ) Computing the forward pass ( Eq . 2 ) requires the knowledge of the exact value of z . The latent code can be computed by employing a parameterized encoder . Similar to discrete-time analogues , we propose to employ a non-causal model as an encoder .

Summary: The paper describes a way of representing online handwriting data, typically seen as a discrete sequence, in a continuous space, using neural ODEs. The learned representation allows sampling from and interpolation in the latent space. The results in the paper are compared (mostly qualitatively) to autoencoder-learned representations based on representing data as discrete sequences, Bezier curves, and implicit surface representation from differentable geometrty.