Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a novel approach to represent chirographic drawing data with Neural ODE and demonstrates its effectiveness by comparing with the standard baseline Seq2Seq model.
main_review: + new domain using Neural ODE
+ well written
+ well-thought domain-specific implementation tricks to adopt Neural ODE
+ evaluated with multiple drawing datasets
- limited domain experiment
- incomplete ablation study
- heavy dependence on the original Neural ODE work

The paper presented a list of implementation tricks to improve the performance of their Neural-ODE-enabled sequential model. The ideas of using the Perlin Noise and periodic activation functions are up-to-date and show their effectiveness to a certain degree.

Also I greatly appreciate the authors providing the details for working with the Multi-stroke format data, not just with Full-sequence format. 

The paper is very well written, but it was difficult to assess the level of novelty. The authors claim that this is a new continuous-time sequential model, but they only tested their model with drawing datasets, that seems to me that there is a little bit of overstatement. This paper could be improved if the authors provide experimental results in other sequential domains, such as speech. Also, adding to that, it would be most appreciative if the authors could add a list of main contributions when this is compared against Neural ODE, as many ideas are dependent on this previous paper and it made me difficult to easily evaluate the level of contributions.

Finally, it could have been better if other new methods that work with sequential data, such as transformer-enabled models, are added to the baseline. I found it very odd that the authors did not include any of those methods for comparison.
summary_of_the_review: Overall, the paper is very well-written and shows fascinating results with continuous-time seq2seq model. 
However there are still some missing components in the paper, and more clarification needed to distinguish this method from the Neural ODE work.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: By introducing the techniques from neural ODEs on chirographic data, the authors were able to train a first continuous-time generative model. New observations that are only available on continuous-time generative models are made.
main_review: ### Strengths
- The newly proposed continuous-time generative model is novel, and several intriguing observations are made. Although the chirographic data itself might have limited application, it served as a great benchmark for testing such a continuous-time generative model.
- The paper is clearly written and well organized overall.
- Considerations of the application on different data domain makes the implication broader.

### Weaknesses
- The tradeoff between computational complexity and representation power induced by using Neural ODEs is not directly handled. To mitigate this, the authors might want to simply share some information about the train/prediction cost and compare them with existing methods.
- For the sinusoidal activation functions, the frequency needs to be manually selected.

### Questions
- In the context of the multi-stroke dataset in Appendix C, what exactly are linear transformations u, v? Are they learned?

### Minor typos found
- At the end of the Introduction part: A surprising 'properly' of Seq2seq... -> property
- Section 4.2: 'Nautral' Spline curve -> Natural
- Section 5.1: 'Range'-Kutta method -> Runge

summary_of_the_review: This work successfully proposed the first continuous-time generative model and discovered intriguing observations. The possibility of different applications might interest a wide range of ICLR audiences, and hence I recommend accepting the paper.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a new model, called SketchODE, for learning representations of sketches using neural ODEs. Specifically, the authors parameterize hand drawn strokes as solutions of ODEs and build an auto encoder like framework for learning the vector fields of these ODEs from data. 

The decoder is modelled with a second order neural ODE acting on an augmented state (which allows for self overlapping trajectories which is necessary for handwritten data) that includes the stroke itself as well as an underlying hidden state. The decoder is made conditional by passing a latent vector to the model, which is mapped both to the initial state of the ODE as well as the dynamics function of the ODE. Varying this latent vector and solving the ODE then gives rise to different handwritten patterns.

The encoder is parameterized by a neural CDE which takes as input some ground truth trajectory and returns a new state which is mapped to the latent vector. The model is then trained end to end using a reconstruction loss between the ground truth and reconstructed trajectories.

In addition, the authors show that two additional tricks can further improve performance: using sine activation functions in the MLPs parameterizing the vector fields as well as using perlin noise to create (continuous) augmentations of the data.

The authors then discuss and demonstrate various compelling properties of their model. This includes the existence of a latent space (which allows for interpolation of handwritten data) as well as one shot learning capabilities for new classes of data.

The main contributions of the paper in my eyes are then:
- Introducing an interesting continuous neural model of handwritten data
- Demonstrating compelling properties of this continuous model over traditional discrete models
- Introducing the VectorMNIST dataset, which I believe could be useful for other researchers
- Interesting experiments on various handwritten datasets



main_review: Overall, I think this is a strong paper which introduces an interesting model and demonstrates various compelling properties of this model through a series of experiments on several handwritten datasets. The main strengths and weaknesses (in my eyes) are described below.

**Strengths**:
- I think the main strength of the paper is the demonstration of the various compelling properties that arise from modeling the handwritten data continuously. In particular, I really like the experiments showing that by simply adding noise to the latent vector we can generate conditional samples from the model, due to the continuous nature of the setup (which would break down in the autoregressive RNN case). The latent interpolation experiments are also very nice. The one shot learning capabilities demonstrated on VectorMNIST are also compelling.
- The model and ideas are very well motivated. Handwriting and drawing are inherently continuous, so it makes sense to model this data continuously. The authors do a good job of demonstrating this, both by contrasting it with previous work and by showing through experiment that this works well.
- The paper is generally well written and the figures are great.
- While the model is somewhat complicated and has lots of moving parts, each part is generally quite well motivated.
- The experiments are generally thorough and well done (on 3 fairly different datasets). The visualizations of the results are great. However, as discussed in the weaknesses section, it would be nice to have more samples/examples from the model (particularly on the more complicated datasets), as well as having error bars.

**Weaknesses**:
- It would be nice to have more samples and results from the model in the appendix. The majority of the results both in the main paper and appendix are shown on VectorMNIST. It would be nice to have e.g. more examples of latent interpolations and conditional sampling on Quick Draw and DiDi. At this point, since there are so few examples they can feel cherrypicked.
- The authors do not discuss the large amount of relevant work that has been done around using neural ODEs for physics (see Hamiltonian neural networks and Lagrangian neural networks and the numerous follow up works since then). Sentences like “there has been little applied work using neural ODEs” and “parameterized ODE dynamics models have largely been treated as a replacement for ResNets where the intermediate states are of little importance” are misleading. As far as I can tell, the authors do not cite a single neural ODE + physics paper which both use the intermediate states and are applications of neural ODEs.
- The authors overclaim their contributions in certain parts of the paper. This is already a great paper, so it is not necessary to do this. For example, “we increase the flexibility of Neural ODE models by introducing a new class of parameterized dynamics functions”. As far as I can tell the authors simply replace tanh/relu MLPs with SIREN (a pre-existing model), which effectively corresponds to changing an activation function. The properties of SIREN models are already well described in other papers, so I think it is a stretch to call this “introducing a new class of parameterized dynamics functions”. In addition sentences like “Learning meaningful representations for chirographic drawing data such as sketches, handwriting, and flowcharts is a gateway for understanding and emulating human creative expression” seem unnecessarily grandiose and don’t provide much information.
- It would be nice to have a more thorough discussion of limitations. As far as I can tell, the only discussion of limitations is one sentence in the conclusion mentioning the computational complexity of the method. It would be nice to have some more thorough discussion of this or ideally a quantitative analysis of how much slower/expensive it is.
- On page 1, the notation h[n] and s[n] is used without being introduced. It only becomes clear what this is later in the paper.
- There are quite a few typos in the paper. I think it would be worth going over the paper another time to clean these up.

**(Some) Typos**:
- Page 3, top: “as generic functions have not yet been” -> “as generic functions have not yet been”
- Page 3, top: “Neural ODE framework” -> “The neural ODE framework”
- Page 3, bottom: “with Neural ODE as in Eq 1” -> “With a Neural ODE as in Eq 1”
- Page 4, middle: “computing the forward pass require” ->  “computing the forward pass requires”
- Page 4, middle: “a non-causal model as encoder” -> “a non-causal model as an encoder”

*Note*: While I am quite familiar with neural ODEs, I am less familiar with the literature around learning representations for handwritten data, and have therefore put a confidence score of 3 on the review. The authors seem to provide a thorough discussion of related work for handwritten data, but it is possible that there are papers that have been missed in that area which I do not know about.

summary_of_the_review: This paper introduces a new continuous time model for learning representations of handwritten data. The paper is well motivated and well written and demonstrates various compelling properties of the proposed model through thorough experiments. While the paper has a few minor weaknesses, I believe it deserves to be accepted.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper describes a way of representing online handwriting data, typically seen as a discrete sequence, in a continuous space, using neural ODEs. The learned representation allows sampling from and interpolation in the latent space. The results in the paper are compared (mostly qualitatively) to autoencoder-learned representations based on representing data as discrete sequences, Bezier curves, and implicit surface representation from differentable geometrty.
main_review: The authors propose an approach of representing online handwriting data in the continuous space. One of the central claims made by the authors is that models that use discrete representation need to spend their learning capacity for learning both the global structure and the temporal continuity (basically, the sampling method), while the models using continuous representation can use their full capacity to represent the global structure only, thus becoming more data-efficient.

Authors represent the continuous state of the pen $s(t)$ (which can include either only coordinates, if a single stroke is encoded, or both the coordinates and the pen state, if several strokes are encoded together) as the observations following the hidden process h(t). Authors also capture the "velocity" of the pen tip in their model. To obtain the ground truth data for it, they fit spline curves to the existing data points.

The encoder of the proposed model used Neural CDEs and computes the hidden state by integrating the input trajectory with a kernel that is a parametrized neural network. The decoder of the proposed model uses Neural ODEs to model the velocity, which can then be turned into the recostruction of the trajectory itself. Both during encoding and decoding, the forawrd pass involves computing an ODE solution, which is done by the black-box ODE solver, using Range-Kutta method. The loss is a point-wise loss (Huber loss or smooth L1) between the points of real and recostructed trajectory, with different sampling frequencies according to the dataset. Authors provide extensive details on the training procedure.

Authors evaluate their approach on QuickDraw dataset of handwritten sketches, DiDi dataset of handwritten diagrams, and the collected small-scale dataset of vectorized MNIST digits. Authors suggest a data augmentation technique to increase the amount of training data, based on adding continuous noise to the continuous trajectory, rather than by adding independent Gaussian noise to the points.

Authors perform the following evaluations:
* Accuracy of the trained recognition CNN on the reconstructed data from 4 models: RNN-RNN seq2seq model trained on raw points data, the one proposed by the authors (SketchODE), and two others: CoSE which encodes a discrete sequence into an implicit representation from which the points of the recostructed curve can be sampled by providing a $t\in[0,1]$, and BezierSketch, which operates on Bezier curves parameters as inputs / outputs to the autoencoder model. CoSE achieves the highest accuracy, closely followed by RNN-RNN, and SketchODE.

**Concern** The Table 1 describing the recognition accuracy of the classifier trained on the real samples, on the reconstructed samples, is meaningless without knowing what is the recognition accuracy of the said classifier on the real data. If it is, for example, lower than 93%, that means that the autoencoder models suffer from a mode collapse and are only able to generate some modes of the training data distributions, that are more recognizable, than the whole distribution in general.

* Authors perform visual comparison to RNN-RNN in terms of the quality of reconstructions, data efficiency, and the ability to interpolate in the latent space. They show favourable comparisons in all cases.

**Concern** Authors compare to the model that operates on the discrete sequence of points, but ignore the comparison to BezierSketch, and, more importantly, CoSE, which are able to provide the continuous reconstructions, and which would have been much fairer comparisons.

**Concern** Authors talk about data efficiency, but don't mention whether they use the same data augmentation when training RNN-RNN (adding Perlin noise), or is that only applicable when training SketchODE (and, if that is the case, whether an independent Gaussian noise data augmentation is applied to RNN-RNN). Authors also don't mention the effect of the data augmentation on the performance of their model.

**Concern** Authors show  the interpolation in latent space between two novel objects, and compare the quality to RNN-RNN. While this indeed indicates the smoothness of the learned latent space, but, (1) this is incomplete without comparisons to methods that allow outputting a continuous curves, and (2) interpolation in the vector space of curves would have likely produced the results similar to the ones obtained by the authors.

Authors show an interesting emergent property of their work, namely, by controlling the capacity of the model decoder, they can control the amount of "complexity" or "high frequency" in their reconstructions.

summary_of_the_review: Authors propose a way of encoding online handwriting data in a continuous space representation by using Neural ODEs, and show interesting properties of the learned model. This is a novel idea interesting to wider audience.

Their main weakness of the paper, however, is the experimental section, and based on its' current content I can not recommend the paper for acceptance. 
* Most importantly, the comparisons are only done to an approach that operates purely on the discrete sequences of points, rather than the ones that, similar to the approach outlined by the authors, allow sampling a continuous trajectory as the output of their decoder (BezierSketch, CoSE), and whether the provided approach provides capabilities beyond those is an open question. Combined with the complexity of the training requiring ODE solvers, it is not clear whether the proposed approach provides any advantages.
* The ablation study is virtually non-existent and does not allow to understand the effects of some of the decisions made by the authors: how does continuous data augmentation affect the quality of the results, rather than the validation error? What are the trade-offs between using a single-stroke representation vs multiple strokes representation? How does the multiple strokes model behave with sequences containing longer than 5 strokes, and how does a single stroke model handle long strokes?
review_title: null

Meta-review: The SketchODE submission is a continuously-valued model for chirographic drawing data such as handwritten digits or sketches. It relies on variational sequence-to-sequence model where the latent code z is a global encoding of the drawing dynamical, and contains a neural controlled differential equation encoder to encode a discrete 2D drawing sequence s, and an augmented neural ODE decoder (conditional on the latent code z) to model both the first-order dynamics both of the drawing velocity and of the pen state (effectively modelling second order dynamics on the pen position). The model enables to sample sketches by sampling latent codes, as well as to interpolate between two latent codes, and is evaluated on VectorMNIST (a new task), QuickDraw sketches, and DiDi schematics, where it is compared to discrete RNN-based Seq2Seq and two more recent baselines.

Reviewers praised the idea of using continuously-valued Neural ODEs for drawing, compelling properties of the model for conditional generation or interpolation, the new VectorMNIST dataset, and the writing. Reviewers had some concerns: overstating the novelty and contribution to general continuous seq2seq given that the evaluation was done only on chirographic drawing tasks (Q3GY, zrrF), some experimental details such as missing ablations, examples from QuickDraw or Didi, or comparisons with transformers (Q3GY, zrrF), clarifications on computational complexity (zrrF, S7jh), situating the work with respect to applications of Neural ODEs to physics (zrrF); most of these concerns were addressed in the rebuttal. Reviewer KvGm had the most concerns about the experimental section, but has increased their score after the discussion with the authors.

There was no discussion among the reviewers, only between the authors and reviewers zrrF and KvGm. After the authors' rebuttal, the scores became 8. 8, 6 and 5, and thus I believe that the paper meets the conference acceptance bar.
Decision: Accept (Poster)