Venue: ICLR 2018 conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The paper aims tackles the problem of generate vectorized sketch drawings by using a RNN-variational autoencoder. Each node is represented with (dx, dy) along with one-hot representation of three different drawing status. A bi-directional LSTM is used to encode latent space in the training stage. Auto-regressive VAE is used for decoding. 

Similar to standard VAEs, log-likelihood has bee used as the data-term and the KL divergence between latent space and Gaussian prior is the regularisation term. 

Pros:
- Good solution to an interesting problem. 
- Very interesting dataset to be released.
- Intensive experiments to validate the performance. 

Cons:
- I am wondering whether the dataset contains biases regarding (dx, dy). In the data collection stage, how were the points lists generated from pen strokes?  Did each points are sampled from same travelling distance or according to the same time interval?  Are there any other potential biases brought because the data collection tools?
- Is log-likelihood a good loss here? Think about the case where the sketch is exactly the same but just more points are densely sampled along the pen stroke. How do you deal with this case?
- Does the dataset contain more meta-info that could be used for other tasks beyond generation, e.g. segmentation, classification, identification, etc.? 
review_title: Interesting problem and good approach to solve it.

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper introduces a neural network architecture for generating sketch drawings. The authors propose that this is particularly interesting over generating pixel data as it emphasises more human concepts. I agree. The contribution of this paper of this paper is two-fold. Firstly, the paper introduces a large sketch dataset that future papers can rely on. Secondly, the paper introduces the model for generating sketch drawings.

The model is inspired by the variational autoencoder. However, the proposed method departs from the theory that justifies the variational autoencoder. I believe the following things would be interesting points to discuss / follow up:
- The paper preliminarily investigates the influence of the KL regularisation term on a validation data likelihood. It seems to have a negative impact for the range of values that are discussed. However, I would expect there to be an optimum. Does the KL term help prevent overfitting at some stage? Answering this question may help understand what influence variational inference has on this model.
- The decoder model has randomness injected in it at every stage of the RNN. Because of this, the latent state actually encodes a distribution over drawings, rather than a single drawing. It seems plausible that this is one of the reasons that the model cannot obtain a high likelihood with a high KL regularisation term. Would it help to rephrase the model to make the mapping from latent representation to drawing more deterministic? This definitely would bring it closer to the way the VAE was originally introduced.
- The unconditional generative model *only* relies on the "injected randomness" for generating drawings, as the initial state is initialised to 0. This also is not in the spirit of the original VAE, where unconditional generation involves sampling from the prior over the latent space.

I believe the design choices made by the authors to be valid in order to get things to work. But it would be interesting to see why a more straightforward application of theory perhaps *doesn't* work as well (or whether it works better). This would help interesting applications inform what is wrong with current theoretical views.

Overall, I would argue that this paper is a clear accept.
review_title: Exciting application that employs deep learning tricks to get exciting results

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper presents both a novel large dataset of sketches and a new rnn architecture to generate new sketches.

+ new and large dataset
+ novel algorithm
+ well written
- no evaluation of dataset
- virtually no evaluation of algorithm
- no baselines or comparison

The paper is well written, and easy to follow. The presented algorithm sketch-rnn seems novel and significantly different from prior work.
In addition, the authors collected the largest sketch dataset, I know of. This is exciting as it could significantly push the state of the art in sketch understanding and generation. 

Unfortunately the evaluation falls short. If the authors were to push for their novel algorithm, I'd have expected them to compare to prior state of the art on standard metrics, ablate their algorithm to show that each component is needed, and show where their algorithm shines and where it falls short.
For ablation, the bare minimum includes: removing the forward and/or reverse encoder and seeing performance drop. Remove the variational component, and phrasing it simply as an auto-encoder. Table 1 is good, but not sufficient. Training loss alone likely does not capture the quality of a sketch.
A comparison the Graves 2013 is absolutely required, more comparisons are desired.
Finally, it would be nice to see where the algorithm falls short, and where there is room for improvement.

If the authors wish to push their dataset, it would help to first evaluate the quality of the dataset. For example, how well do humans classify these sketches? How diverse are the sketches? Are there any obvious modes? Does the discretization into strokes matter?
Additionally, the authors should present a few standard evaluation metrics they would like to compare algorithms on? Are there any good automated metrics, and how well do they correspond to human judgement?

In summary, I'm both excited about the dataset and new architecture, but at the same time the authors missed a huge opportunity by not establishing proper baselines, evaluating their algorithm, and pushing for a standardized evaluation protocol for their dataset. I recommend the authors to decide if they want to present a new algorithm, or a new dataset and focus on a proper evaluation.
review_title: review

Meta-review: This work presents a RNN tailored to generate sketch drawings. The model has novel elements and advances specific to the considered task, and allows for free generation as well as generation with (partial) input. The results are very satisfactory. Importantly, as part of this work a large dataset of sketch drawings is released. The only negative aspect is the insufficient evaluation, as pointed out by R1 who points out the need for baselines and evaluation metrics. R1’s concerns have been acknowledged by the authors but not really addressed in the revision. Still, this is a very interesting contribution.
Decision: Accept (Poster)