Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: - The paper proposes an unsupervised neuro-symbolic generative model for general-purpose pen-drawing/doodling frames as a sequential composition of strokes. 
- Analyze the model and explore its benefits in terms of interpretability and out-of-distribution generalization. 

strengths_and_weaknesses: **Strengths**
- **Technical details**: The paper does a great job in describing the model in a formal, precise and detailed manner.
- **Writing quality**: The writing quality is good and the model figure is clear and helpful.
- **Experiments**: The paper extensively covers a variety of experiments measuring generalization across both different tasks and datasets, provides a lot of helpful visualizations and also performs ablations.
- **Unsupervised**: The model achieves SOTA performance compared to other neuro-symbolic approaches even though it doesn’t rely on strong supervision or augmentations of the data.
- **Out of distribution generalization**: The general-purpose framing of the approach allows it to generalize out of the training distribution (MNIST) to new characters (Japanese) and even doodles (car, bike) and to a greater number of strokes. It does better than AIR.
- **Interpretability**: The model generates doodles in a sequential easy-to-visualize manner, decomposing them in mostly a natural manner – based on looking at the few shown examples, similar in some cases to how a person might approach the drawing task (although in other cases the decompositions are less likely). 
- **Generative capabilities**: The model can create new examples given a new doodle, and can complete a partial doodle into diverse results. 

**Weaknesses**
- **Task**: The model explored by the paper is very specific to the particular task (decomposing doodles to strokes, formulating a stroke as a controlled curves that could be adjusted through spatial transformations), which is potentially a bit narrow in itself. It’s unclear to me what is the potential of the approach to scale to broader domains. It could be interesting to explore visual compositionality in a more general manner, but that’s not studied in the paper.
- **Model complexity**: the model is quite complex, with 5 different models and subparts:  having generation and recognition models, with the former consisting of sampling the layout, then the strokes, then performing an affine transformation on them (rendering), then compositing them. I think it would be good to both add more intuitive explanation about this process, its structure, and why it is a good way to approach the task. And if there could be simplifications to the approach that would also be great. 
- **Idea presentation**: I think the presentation could be improved to highlight more the big picture, the motivation and the takeaways. A good amount of background context, related work and overall motivation are being discussed, as well as the limitations and ethical considerations.

Considering both strengths and weaknesses, I originally was in terms of scoring between 6 and 7, leaning to 7 due to technical contribution and formal presentation and to 6 due to the potentially bit narrow scope of the task, but then due to the great evaluation section I converged at 7.
questions: - **AIR**: The first paragraph talks about decomposing to “what” and “how” etc. I think the AIR paper should be cited at this point to give it the credit of originating these ideas.
- **Affine transformations**: Is it possible to decompose the doodles to general sub-curves that will be directly sampled relatively to the positions of the previous strokes, such that there won’t be a need to apply transformations on each of them? If I understand correctly, it may help simplify the model.
- **Related work**: I recommend moving the related work section to the beginning of the paper after the intro rather than before the end. 
- **Figure**: In figure 3 right it’s unclear to me why it says MNIST often has 10 strokes while quickdraw has 2? Are these labels swapped or I misunderstood something? 

limitations: Yea but only in the supplementary. I recommend making at least a brief version in the main paper.
review_title: null

final_score: rating: 2: Strong Reject: For instance, a paper with major technical flaws, and/or poor evaluation, limited impact, poor reproducibility and mostly unaddressed ethical considerations.
review_content: summary: the work models the drawing, especially, consisting of simple strokes, as a sequential generation problem. The proposed approach leveraged the spatial transformer network to learn and decouple the variance between samples originating from spatial transformations. The generative approach is applied in two task: generating new plots, and one-shot classification. 
strengths_and_weaknesses: Strengths
+ plenty of qualitative results and benchmark results are presented. the advantage is highlighted clearly

Weakness:
- AIR is a baseline from 2016, which is pretty outdated. The method proposed seems to be like an STN version of AIR, or in other way, a sequential version of STN. 
- The impact of the work seems pretty limited. STN and AIR both explored if they could apply to more realistic dataset like image classification, object detection, etc. while the proposed approach so far proved to only work on stroke-based drawing. 
questions: How did the authors choose the baseline models to compare? is every model in table 1 generative? Why they are not compared with general classification models?


limitations: The limitation was discussed outside the main submission. The approach handles a very specific task and the generalization to other tasks are limited.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a neuro-symbolic generative model for stroke based drawing. Unlike prior work in neuro-symbolic generative models, the method doesn’t require explicit supervision. It consists of two parts: 1) A generative model that models the distribution over strokes (sampled from Bezier splines) and their layouts. It is also responsible of rendering the strokes on the canvas in differentiable manner (referred as guided execution). 2) A recognition model that conditions on the target image and infers where to place the strokes. The efficacy of method is demonstrated on two sets of tasks: 1) Generalisation of stroke based reconstruction across datasets. 2) Generalisation across tasks which depict the usefulness of the representation learned.
strengths_and_weaknesses: **Strengths**
- The paper is largely well written and the novelty is clear.
- Shows impressive performance for out-of-distribution characters reconstruction, unconditional generation, conditional generation, and one-shot classification.

**Weaknesses**

Although there are no major concerns, I would appreciate if authors could address the following points.

- A graphical representation of the generative model would make the generative process easier to understand.
- The paper needs to connect certain portions better. E.g how does generative model complements recognition model or vice versa? or how does the shared hidden representations $h_{t}^{l}$ and $h_{t}^{s}$ help recognition and generation? 
- The model does not seem to be a fully generative one. I am probably confused here because of equation 1, as characters generation doesn’t seem to be conditioned on the initial states of layout, stroke and on/off variables. See my related question below.
- I liked the transfer of stroke-based reconstructions across datasets experiments. However, I think it’s one-shot transfer instead of zero-shot as the model makes use of the single sample to guide reconstruction. See my related question below.
- Moreover, some terms are introduced without explanations for e.g.  guided execution and bezier splines.
questions: - What are the prior probabilities of the initial states of the variables corresponding to layout, stroke, on/off variable and image itself i.e. $p(l_0)$, $p(o_0)$, $p(s_0)$ and $p(x_0)$. Shouldn’t they be a part of Eq. 1?
- (Related to above question) For unconditional generation, how do you sample from the sequential prior? E.g. is the initial state value is sampled from Gaussian distribution?
- What does $\textit{m}$ and $\textit{k}$ correspond to in equation 2 and 3? Are they specific to GMM components weights?
- Could Dth order Bézier splines limit the creation of certain characters/shapes? Or introduce a bias for certain strokes in generation. What is the criteria of selecting a distribution of these splines?
- What does differentiable renderer refer to? Is it just a simple affine transformation or some intricate function of putting control points on the canvas?
- It wasn’t clear what guided execution referred to in the paper. I had to read it again to figure out it’s the differentiable rendering. 
- The binary latent variable has some continuous relaxation? The behaviour of Bernoulli distribution (equation 4) can be random in the beginning. Does it affect the training?
- Given the figure 1, shouldn’t the posterior be conditioned on $\Delta x_t$, in L.H.S of equation 6?
- Would a comparison with MWS (https://arxiv.org/pdf/2007.03132.pdf) and another part based generative model (https://arxiv.org/abs/1906.06818) make sense for generalization across datasets experiment? I would recommend to mention the second paper in the related work section.
- Can the transfer of stroke-based reconstructions across datasets be truly called zero-shot as the recognition model have access to the image (x) from OOD dataset? The image from OOD dataset guides the reconstruction (via recognition part of the method) irrespective of whether training was explicitly done on this new dataset or not. This, in my opinion, is one-shot transfer. Curious to know what authors think about this.
- Typo on line 36,  -> model or approaches
limitations: There doesn't seem to be any obvious negative societal impact of this work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a generative model for images (drawings or characters) parameterized by sequences of strokes, and a recognition model for inferring the stroke sequence from an image, which are trained jointly by optimizing a variational lower bound.  The generative model makes use of a Bezier-spline stroke model with control points provided by a recurrent convolutional neural network, and a recurrent spatial transformer network used to frame the next stroke.  The method can be used to reconstruct images from character and drawing datasets, generalize across datasets, generate new characters (hence drawing out of distribution), and to perform one-shot character classification.
strengths_and_weaknesses: ## Originality
This method highly resembles other approaches in the literature, e.g. splines used as drawing elements (Mihai and Hare 2013), use of variational objective (Tomczak and Welling 2018), recursive image generation (Feinman and Lake 2021), or use of spatial transformer network within a character recognition model (VHE, Hewitt et al. 2018).  The combination of recursive image generation with a variational objective was previously explored by AIR (Eslami et al. 2016), while the stroke model resembles GNS (Feinman and Lake 2021).  DooD resembles a fusion of AIR and GNS, utilizing a combination of variational objective with a stroke model, and trained on images rather than stroke sequences.  It may be the first recursive image sketching model that uses spatial transformer networks as part of its composition process.
## Quality
Some strengths of the paper are the evaluations on multiple datasets, including cross-dataset transfer, and variety of evaluation tasks (corresponding to the Omniglot challenge).  It is also interesting that the paper investigates the internal representation of the method compared to similar methods.  The contribution of the recurrent structure of the model is demonstrated by an ablation study.  The method qualitatively outperforms baselines in terms of quality of image reconstruction, but the difference in performance is not quantified.

Some weaknesses of the paper are that many of the modeling choices are not well-motivated.  In particular, the choice to use Laplace noise which appears to accumulate over steps (see below for clarity issues) seems unusual and possibly a poor fit to the actual image distribution of human drawings.  The use of only one real baseline (AIR) seems weak, especially since AIR was not developed specifically for line drawings.
## Clarity
The grammar and organization of the paper is good.  However, the details of the method are not described well.  The stroke and rendering components of the model are particularly hard to understand since key details are missing from the main text, and the notations used in the appendix are not defined in terms of the notation in the main text. I am sympathetic to the difficulty of explaining such a complicated method in the page limit provided, so I am personally OK with putting more details in the appendix.  If the clarity of the methods could be improved, I would consider raising my score, which is currently "reject" because of lack of reproducibility.  I am of the opinion that for the paper to be publishable, it needs to contain a complete description of the method details (either in the main text or the supplement), and not just a code submission.  That said, well-documented code would help the reproducibility of the paper even more in conjunction with a more complete description of the method.
## Significance
This paper appears to present a method for modeling human line drawings and handwritten characters from static images which is claimed to have superior cross-domain generalization, computational efficiency and interpretability compared to previous approaches.  It appears that the motivation of the work is to obtain better models of handwritten characters and drawings for computer vision, rather than modeling human drawing.  The interpretability of the method does seem to be superior to AIR, and while methods not based on recursive drawing are not evaluated in terms of interpretability, it is likely that they would suffer from even poorer interpretability.  The method is evaluated against GNS, BPL, and VHE which require stroke-level data.  However, a more relevant comparison would be against one-shot image classification models that are trained on static images.

questions: * Given the claim that DooD stands out compared to related methods in terms of computational cost, is there any evidence to establish its computational efficiency in practice?
* Does DooD achieve superior one-shot classification compared to other character recognition models based on static images?
* Given that GNS (Feinman and Lake, 2021) can already 'draw out of distribution', what is the motivation for the naming of DooD?
* Line 93: Why is the decision to stop made before composing the current stroke (rather than afterward)?  Doesn't this mean wasting the computational effort to draw the current stroke if it isn't even used?
* Line 96: What is the justification for this Laplace noise model? Why is the scale of tha Laplace fixed? What is the range of the pixel intensities, and is the Laplace distribution appropriate given that range?  Why are the pixel intensities assumed to be conditionally independent given the composited image?  Does the addition of Laplace noise accumulate over time t?  Doesn't this mean that the variance of pixels from earlier strokes is inflated?  Why?
* Equation 6: What is the range for the index t?
* Supplement B.1.  The notation used here (e.g. equations 10-14) differ from the notation used in the paper.  How are the terms in equations 1-6 defined in terms of these quantities?
* Line 119: How is beta tuned?

limitations: Yes.
review_title: null

Meta-review: metareview: The paper presents DooD, an unsupervised approach for stroke-based generation of line drawings and handwritten characters, which the authors show outperforms other methods in generalization and interpretability. The reviewers agree that this is an innovative and useful contribution, and is well-expressed in the paper. Some concerns are raised related to the insights that one can draw from this domain to other forms of image generation; however, these objections are not fatal, in my opinion. Note that in a private communication with me, the only reviewer not recommending acceptance wrote: "I would keep my score, though I won't be disappointed if the paper will be accepted."

I recommend acceptance of the paper.
confidence: Certain
Decision: Accept