Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: -- define cascade errors when you first use the phrase
-- basic english grammar could be fixed but is not interfering with understanding
-- what is the early stopping criterion in Alg 1?
-- did you try any other values for the initial threshold and decay factor? 
-- At the end of Section 4.1 DROP: ", but not the same." - I can't parse what this last clause is supposed to mean.
--the diff sum example in table 2 is confusing; the program appears to sum up the numbers but the result is a subtraction without a sum operation in it. Would be clearer to show the sum in the result line as well rather than distribute the subtraction. Also, shouldn't it be diff(9, sum(10, 12))?
-- I think you should pull at least some commentary about the constant used in Table 3 from Appendix B and include it in the main paper (or at least mention Appendix B is the place to look). Can you add a table in an appendix showing the complete list of operators used? 
-- Nice results in Table 4 on the dev set. Are there Test set results as well?
-- The organization of the Baselines 4.3 section and the Results 4.4 is confusing. For example, you mention that you test different variants of NeRd, operator variants, and mathqa, but then the results are not mentioned for these experiments until the next page. I found myself immediately looking for the numbers/results when you introduce the experiment. I would pair your experiment description with the results rather than grouping all experiment descriptions and then grouping all results, especially when the order of the experiment descriptions does not match the order of the results presented. For example, in baselines you discuss training variants and then operators. Then in Results you discuss operators before variants. It is too disconnected and makes the reader jump around a bunch. Same goes for the drop baselines where you mention a bunch of models, and I would prefer the Results/discussion paired with each one, rather than having to wait for it down below.
-- Overall it seems like a solid work; good empirical results showing improvements of each purported contribution. The model itself is a relatively simple construction of basic component, but the combination with the DSL is intuitive and makes sense. I don't think the novelty in model here is the main selling point anyways; the training variants and the demonstration of how well a DSL approach can do combined with previously introduced methods.
-- I find the model description to be slightly unclear. In Fig 1 for example there is an arrow that connects passage to compositional programs. What does that arrow represent? I think you should elaborate on how the attention over the encoded text interacts with the attention over previously generated tokens. Equations would make this far more explicit as is I am left with a lot of questions on how to implement your model. Maybe you can add to your appendix? Or release your code? That is mentioned either.
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper discusses an extended DSL language for answering complex questions from text and adding data augmentation as well as weak supervision for training an encoder/decoder model where the encoder is a language model and decoder a program synthesis machine generating instructions using the DSL. They show interesting results on two datasets requiring symbolic reasoning for answering the questions.

Overall, I like the paper and I think it contains simple extensions to previous methods referenced in the paper enabling them to work well on these datasets.

A few comments:

1 - Would be interesting to see the performance of the weak supervision on these datasets. In other words, if the heuristics are designed to provide noisy instruction sets for training, we need to see the performance of those on these datasets to determine if the models are generalising beyond those heuristics or they perform at the same level which then may mean we don't need the model.

2 - From Tab 4, it seems the largest improvements are due to span-selection cases as a result of adding span operators to the DSL. A deep dive on this would be a great insight (in addition to performance improvement statements on page 7).

3 - Since the span and value operators require indices to the text location, could you please clarify in the text how that is done? Do LSTMs output the indices or are you selecting from a preselection spans as part of preprocessing?


review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: This paper presents a semantic parser that operates over passages of text instead of a structured data source.  This is the first time anyone has demonstrated such a semantic parser (Siva Reddy and several others have essentially used unstructured text as an information source for a semantic parser, similar to OpenIE methods, but this is qualitatively different).  The key insight is to let the semantic parser point to locations in the text that can be used in further symbolic operations.  This is excellent work, and it should definitely be accepted.  I have a ton of questions about this method, but they are good questions.  The rest of this review focuses on things that I thought could be more clear, or that raise new questions, and might sound negative.  Please understand them, however, in terms of my overall score and what I said above.

The three claimed contributions are (1) better numbers, (2) better compositionality / domain applicability, and (3) better interpretability.

(2) and (3) sound a bit like overclaiming in the introduction to me, as there isn't a whole lot of nested composition in the language used by NeRd, and the BERT calculator in principle is almost as compositional and interpretable (also, e.g., NAQANet can add and subtract an arbitrary number of numbers, also, and it tells you which ones they are, just as NeRd does).  Later in the paper the specifics of those claims are made more clear, and while they are justified, they are very narrow claims.  To me, someone who is intimately familiar with this research area, the key contributions (the things that I learned) are (1) using passage-span and key-value predicates actually works, (2) how much difference hard EM and thresholding make, and (3) the data augmentation in this work is pretty clever.  (2) was intuitively clear to me after seeing Dasigi's iterative search paper and Min's hard EM paper, but the difference in results presented here is pretty striking.

Compositionality:

The authors claim that their method is compositional and domain agnostic, while all previous methods had hand-crafted modules for specific question types.  However, I see little reason to believe there's much of a difference here.  You also defined operations that are tailored to the dataset, and are basically identical to the operations that others have used.  I see no evidence that NeRd actually generalizes to program types that are beyond what is captured by other methods.  It's possible that this happens, but there is no evaluation that discusses this, and from all of the examples I'm led to believe that this is basically also just learning a few program templates, the same ones learned by previous methods.  With the weak supervision that you have, are you actually able to find more complex programs during your search?  Some kind of demonstration of actual compositionality on the more complex questions in DROP would make a very strong argument for your claims; without that, they ring a little hollow.

Interpretability:

The use of passage-span as a predicate is really interesting, and it raises a lot of questions.  This predicate lets the model shortcut any interpretable reasoning and do operations entirely inside the encoder/parser.  For example, your first example in table 2 ostensibly requires filtering the numbers in the passage to those that are percentages associated with groups, then filtering them again to those where the percentage is larger than 16, then returning the associated groups. But your method jumps straight to returning a set of passage spans.  This is hardly interpretable.  (In fairness, no prior method provides interpretable reasoning for this kind of operation either.)  But the fact that you have this predicate lets the model do these filters and greater-than comparisons inside the network in an opaque way, while also getting interpretable operations for some questions (table 5 is further confirmation of this, and of the fact that you probably are not capturing many of more the complex, compositional questions in DROP).  But how does the network decide which to do?  Any argmax or max question, and many count questions, could be answered by passage-span alone.  With only weak supervision, and with the parser having the ability to shortcut these more interpretable operations, how often are you actually getting the interpretable one, and what's causing the model to choose it?

Similarly, how often does an argmax or a max operation actually operate on the full set that you would expect it to?  Or does it just do the argmax internal to the network, and output only one item as an argument to the argmax?  If the later, this again hurts your claims of better interpretability over prior methods, as the logic is just as opaque as before.  This also seems like a really hard search problem in how you've set up your DSL - what would make your search over programs actually select all of the correct arguments?  Because you're selecting passage spans directly instead of performing some kind of matching operation, you have to have your search select all of the appropriate spans for this to be "interpretable", and not just hiding the logic inside of the network.  But that seems like a totally intractable search.  You found a clever way to get around this for count questions (even though that still implicitly hides a bunch of filtering logic, as noted above), but I don't know how to make it work for maxes and argmaxes.

Another question raised by the passage-span predicate: the more you use bare passage-span programs for training, the more the network learns to put all of its compositional reasoning inside, in an opaque way, instead of giving you interpretable compositionality.  At one extreme, you end up with something like NABERT (or even less compositional), where basically everything is inside the network.  At the other extreme, where you don't have passage-span, you are left with a crippled semantic parser that can't handle most of the questions.  But using the predicate introduces tension in the model between interpretability and flexibility.  How do we resolve this tension?  (This isn't something I expect your paper to address, it's just a really interesting and important question raised by your work.)

Parser:

Prior work has found benefit in using runtime constraints on parser outputs, or grammar-based decoding.  It looks like you are doing neither of those, yet you're able to output specific token indices and number indices in your programs.  Are you really not doing anything special to handle those?  How does the decoder know token indices?  I feel like something must be missing here, or a simple LSTM decoder is more magical than I thought.

Evaluation:

Why only show results on DROP dev, and not on the test set?  It's possible that your higher numbers are because you were better able to overfit to the dev set, which you presumably used during training.  (I don't think that that's likely, but it's a concern that would be easily avoided by evaluating on test.)
review_title: Official Blind Review #2

Meta-review: Main content:

Blind review #1 summarizes it well:

This paper presents a semantic parser that operates over passages of text instead of a structured data source.  This is the first time anyone has demonstrated such a semantic parser (Siva Reddy and several others have essentially used unstructured text as an information source for a semantic parser, similar to OpenIE methods, but this is qualitatively different).  The key insight is to let the semantic parser point to locations in the text that can be used in further symbolic operations.  This is excellent work, and it should definitely be accepted.  I have a ton of questions about this method, but they are good questions.  

--

Discussion:

The reviews all agree on a generally positive assessment, and focus on details that have been addressed, rather than major problems.

--

Recommendation and justification:

This paper should be accepted. Even though novelty in terms of fundamental machine learning components is minimal, but the architecture employing neural models to do symbolic work is a good contribution in a crucial direction (especially in the theme of ICLR).
Decision: Accept (Spotlight)