Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper introduces generative neuro-symbolic modelling, advertised as a probabilistic programming framework in which the distributions are modelled by neural networks. This is a very exciting idea, and well past its time. However, the work discussed here is limited to modelling the drawn characters in the Omniglot dataset, rather than a general framework. It is clear there is not a usable tool that would allow for practical construction of a wide range of probabilistic programs, from the fact that the paper sticks with a simple model for the Omniglot problem, rather than doing a lot of manipulations on the model, and, more importantly, the fact that the inference is specific to the Omniglot model, and is presumably not very fast, given the description. Thus, while the sales pitch to the paper is very exciting, compared to that, the content itself, which is really just a model of one problem, is a bit disappointing. That was also a property of the earlier Lake et al paper - I never understood what the "framework" was there either. That is not to dismiss this paper. The insight for this problem is that, rather than relying on a database of character primitives, like the previous Bayesian model of the problem, you can sequentially generate characters (types)  non-independently using the expected cast of characters (convolutions and LSTMs - although there are also splines involved, so it's not exactly typical CNN either). The evaluations are not all that impressive, except for the unconditional character generation shown at the beginning, which is clearly outstripping the BPL model in terms of human-like ness. In Table 3, I don't understand why different image sizes are used, and why BPL is not compared. The paper is very clear.

In short, I really like the idea in this paper, and I think it is worth building on. The section in the appendix detailing the future work on 3D object modelling sounds very promising. However, the result itself is not all that significant beyond the idea; the evaluations are not particularly strong, except for the generation of novel characters, which would benefit from a formal evaluation, and perhaps a transfer to a less artificial problem.
review_title: GNS review

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary: This paper presents a generative neuro-symbolic model for learning the task-general representations. The model is inspired by the BPL approach, with less manually-defined prior or rules but more learning-based ingredients. I generally appreciate the ideas and improvements it made. However, compared with BPL, the claims of task-general representation, and neural-symbolic modeling does not match the actual contributions of the work.

Pros
1. The paper is well-written with its motivations,  methods, and corresponding experimental results. It shows a promising direction of learning a representation and model for multiple tasks, \ie, the Omniglot challenge.

2. The formulation of the proposed GNS and the corresponding method of inference provides a well-established and holistic solution for the visual simple concepts learning and inference.

Cons
1. The claimants of human-level visual understanding, and task-general representations are similar to the BPL methods, which is well-known enough. I think the author should put more effort into comparing the BPL and demonstrates the actual improvement compared with it. 

2. The current framework is still under the Bayesian statistical modeling, I don't think it provides a task-general representation since it cannot be transferred to more complex domains and perceptual tasks. Moreover, the symbolic representations and symbolic module in the proposed method are quite trivial. The proposed method does not provide deeper insights into the neural symbolic representations.

3. The causal structures in the model are captures in an autoregressive way, which is similar to lots of existing modules.  I don't see any actual causal problems are solved in the framework.

My main concern is that it put too much space in telling the stories that BPL has already proposed and demonstrated, but not the detailed comparisons with BPL and how the technical contributions and novelty of this specific model can benefit the overall human-level understanding of visual concepts. I would like to raise my scores if the concern is properly addressed.

I raised my scores from 4 to 6 after the author updated their draft and answered my questions. I appreciate their efforts in addressing my concerns and improve the paper. The current version is good enough to be accepted and it also compares with other approaches thoroughly. However, I still feel the symbolic module is too simple in this work and does not distinguish it from other works. 
review_title: Interesting method but the novelty is not enough

final_score: rating: 7: Good paper, accept
review_content: Motivated by few-shot learning challenges such as Omniglot, the authors propose a human-like model for learning how to draw visual concepts that consists of three components: (1) a location model for picking the starting point of the next stroke given the current "canvas", (2) a stroke model that continues an existing stroke, (3) a termination model that decides when to stop drawing.  The three components of this model are trained on example glyphs that are heuristically parsed into strokes.  The authors evaluate performance of this method, called GNS (Generative Neuro-Symbolic) versus more classical program learning approaches based on pre-existing libraries of subroutines and more generic deep neural network architectures.  This approach (1) represents an advance over methods based on pre-defined subroutines in the sense that primitives such as "draw a stroke intersecting an existing stroke" are learned rather than supplied by the programmer and (2) performs much better at few-shot learning of glyphs than generic approaches.

Quality: The idea behind the method makes sense.  The authors evaluate the model on four different tasks, and compare against reasonable baselines.  They also generalize beyond glyphs to learning 3D objects, although results are not shown.  The authors are careful to describe details such preprocessing of the input into splines and candidate parses, although details of the actual architecture do not seem to be given.  The weakest part of the approach is the image model.  It seems to work well enough for performance purposes, but the image distribution involving randomly transforming parts does not seem intuitively plausible as a model of human-like image generation.  As the authors note, this results in their approach underperforming against BPL.  The approach would probably work better if the authors had included stroke variability as a part of the core model.

Clarity:  I had to read the paper a few times to understand the approach.  It would have helped if the authors could better explain the architecture of the various model components.  Besides this issue of vagueness on technical details, the paper is well-written.

Originality:  To my knowledge the use of a separate location and LSTM stroke model for glyphs is novel.  The previous work section is adequate for contextualizing the paper.

Significance:  The paper represents an important approach to mimicking human concept learning.  (I base this on my own intuition; the authors could perhaps make a stronger case for this by including relevant references from the cognitive science literature.) 
Chaining together multiple different neural networks to create a new type of ML algorithm has been a promising source of innovation in deep learning, e.g. AlphaGo combining a policy and evaluation network, GANs combining a generator and discriminator.  This paper is yet another example via combining a location and stroke model for learning glyphs.  On the other hand, the authors may have oversold the significance of the work by claiming that this model represents an example of fundamentally new approach in harmonizing artificial neural networks and symbolic program learning.  I am not convinced that this combination of location+stroke model is doing hierarchical concept learning in the same way that humans do.  If this is not what the authors are claiming, they should clarify this in the main paper.

Pros:
 * novel and plausible model for human-like learning
 * achieves reasonable performance on a variety of one-shot learning tasks
 * qualitatively exhibits human-like performance at free glyph generation (in my opinion)

Cons:
 * the type -> token part of the model seems underdeveloped
 * paper is vague on the architectures used for each of the model components (location, stroke, termination)
 * paper makes a claim that the model is doing symbolic modeling, but the term "symbolic modeling" is not well-defined and there is insufficient analysis showing that the model learns something analogous to existing methods such as BPL

review_title: Plausible model of human visual conceptualization

final_score: rating: 7: Good paper, accept
review_content: ### Summary

In this paper, the authors seek to combine the advantages of symbolic, compositional models and neural approaches, in particular by using "probabilistic programs with neural network subroutines" which should reflect a causal generative process. This is certainly an interesting area to explore. But I'm not convinced that the approach they propose (GNS) lives up to their stated goals.

The concrete model they describe (a generative model of Omniglot characters) seems to be essentially a recurrent neural controller whose outputs are connected to a differentiable stroke-rendering output function, and it seems to me to have very little "symbolic" structure. There is also a hierarchy of "type" and "token" levels (similar to BPL) but this hierarchy makes strong independence assumptions and seems too low-level to capture the true generative process. The authors claim this is a framework for learning task-general representations, but I struggle to see what this framework consists of; their discussion of GNS in section 3 seems highly tailored to the Omniglot task.

The paper is well written and the model itself is described in detail. The experimental results on four Omniglot tasks are strong and quite thorough, and the GNS model does seem to outperform the BPL model and other baselines. The authors also acknowledge some of the shortcomings of their model and describe some interesting possible future extensions.

I lean toward accepting the paper, since it is an interesting approach to the Omniglot tasks and does have some solid contributions. But, at least in it's current state, I feel like the paper falls short of fulfilling its own promises.

### Detailed comments

The introduction and motivation sections are very well written, and put forth a vision of Generative Neuro-Symbolic Modeling (GNS) as a way to combine the advantages of symbolic approaches with learned neural ones. The authors describe how symbolic models can generalize well and behave compositionally, but only when their strong assumptions correctly match the data-generating process; on the other hand, neural models can flexibly learn from data but can struggle to capture compositionality and may not generalize as well. GNS is described as a way to unify these strengths and obtain flexible models that also generalize.

The high-level goals of this approach seem to overlap to some degree with the HOUDINI framework [1], in that both are trying to combine the advantages of neural and symbolic approaches for generalizing to new tasks. There are significant differences, of course, but it is likely worth a brief discussion in the related work section.

The generative model proposed in section 3 seems strange to me in a couple of ways. Like BPL, it is separated into a "type" level, which is meant to capture character concepts, and a "token" level, which is meant to capture individual drawings of each character. In BPL, the "type" level values are structured objects that combine sequences of subparts and link them with discrete relationships, and the "token" level values are collections of strokes that are then rendered. In GNS, however, both the "type" and "token" levels are collections of rendered strokes, and the authors note that the type prior alone already specifies an "auto-regressive density function that can evaluate exact likelihoods of character drawings". The "token" model simply perturbs the strokes from the "type" level with simple 2D Gaussians, independently for each control point in the stroke, and then applies a global affine transformation. The authors claim that this is a causal model of character drawings, but it doesn't match my intuition about what the causal generative process of those drawings should be. In particular, it seems to me that the character concept needs to include the kinds of relational structure captured by BPL, at least implicitly. The representation used by GNS seems too low-level to adequately capture a character concept, and the strong assumption of independent perturbations leads to the loss of certain structural elements (as the authors note in section 5 under "Generating new exemplars").

Given the strong claims made initially, I was really hoping for a more powerful neural token model combined with a more abstract type model. Perhaps this could look something like Sketch-RNN [2] or DRAW, with a "type" prior over an abstract latent space and then a "token" decoder that renders strokes (similar to the proposed GNS type model). It seems to me that that would still having many of the same advantages but would also have the ability to generate more flexible output “tokens” than the GNS model (for instance, it might learn to connect strokes to previous strokes even when those previous strokes are perturbed by noise). The VAE framework might also provide a nice way of generalizing the very-domain-specific bottom-up parse proposal distribution described in the appendix.

More broadly, I get the sense from the introduction that this generative model of sketches is intended to be an instance of a more general framework of some kind (although this isn't particularly obvious, since the sketch-specific model is frequently referred to as simply "GNS" in the paper). But it is not at all clear to me what this framework actually is. On the most general side, is any model that composes neural submodules according to domain knowledge an instance of the framework? This is the impression that I got from the introduction, but it would seem to include a large number of preexisting models; for instance, the DRAW model specifies a structured probabilistic generation process of images. Or perhaps it requires combining neural and domain-specific primitives together into differentiable processes? This would include various papers on differentiable rendering (such as [3] and [4]) or audio synthesis (such as [5]). On the other hand, the conclusion suggests that the GNS framework is only applicable for visual tasks, which is a considerably smaller scope than the introduction suggested. (The authors also include some brief discussion of a GNS model of 3d structures in the appendix, but it seems very different from the model for characters, and I'm not sure what the common thread is.) The paper could be much improved by being more explicit about what the claimed general concept-learning framework actually consists of.

### Questions and suggestions

I was a bit surprised that neural architectures are referred to as "nonparametric". I suppose this is because vastly overparameterized networks behave like nonparametric models, despite having parameters? It might be worth clarifying the meaning here to avoid confusion.

Is equation (4) doing maximum a-posteriori (MAP) inference over the token-level parameters? It would be useful to go into a bit more detail on what this approximation is and why approximating in this way is reasonable.

In section 5 under "Parsing", it wasn't clear to me why having a variety of parses is bad for the first character but desirable for the second one. Is it because that matches what humans do?

Minor issue: I noticed that the authors appear to have used the ICLR 2020 style files instead of the ICLR 2021 style files.

### References

[1] Valkov, Lazar, et al. "Houdini: Lifelong learning as program synthesis." Advances in Neural Information Processing Systems. 2018.

[2] Ha, David, and Douglas Eck. "A neural representation of sketch drawings." arXiv preprint arXiv:1704.03477 (2017).

[3] Li, Tzu-Mao, et al. "Differentiable monte carlo ray tracing through edge sampling." ACM Transactions on Graphics (TOG) 37.6 (2018): 1-11.

[4] Thies, Justus, Michael Zollhöfer, and Matthias Nießner. "Deferred neural rendering: Image synthesis using neural textures." ACM Transactions on Graphics (TOG) 38.4 (2019): 1-12.

[5] Engel, Jesse, et al. "DDSP: Differentiable digital signal processing." arXiv preprint arXiv:2001.04643 (2020).

----
After reading the author responses and updated manuscript, I have raised my score from 6 to 7.
review_title: Interesting approach, but some strange choices, and doesn't quite fulfill its own promises

Meta-review: This paper was reviewed by four experts in the field. Based on the reviewers' feedback, the decision is to recommend the paper for acceptance to ICLR 2021. The reviewers did raise some valuable concerns that should be addressed in the final camera-ready version of the paper. The authors are encouraged to make the necessary changes and include the missing references.
Decision: Accept (Poster)