Venue: ICLR 2018 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper presents a neural architecture for converting natural language queries to SQL statements. The model utilizes a simple typed decoder that chooses to copy either from the question / table or generate a word from a predefined SQL vocabulary. The authors try different methods of aggregating attention for the decoder copy mechanism and find that summing token probabilities works significantly better than alternatives; this result could be useful beyond just Seq2SQL models (e.g., for summarization). Experiments on the WikiSQL dataset demonstrate state-of-the-art results, and detailed ablations measure the impact of each component of the model. Overall, even though the architecture is not very novel, the paper is well-written and the results are strong; as such, I'd recommend the paper for acceptance.

Some questions:
- How can the proposed approach scale to more complex queries (i.e., those not found in WikiSQL)? Could the output grammar be extended to support joins, for instance? As the grammar grows more complex, the typed decoder may start to lose its effectiveness. Some discussion of these issues would be helpful.
- How does the additional preprocessing done by the authors affect the performance of the original baseline system of Zhong et al.? In general, some discussion of the differences in preprocessing between this work and Zhong et al. would be good (do they also use column annotation)?
review_title: review

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper claims to develop a novel method to map natural language queries to SQL. They claim to have the following contributions: 

1. Using a grammar to guide decoding 
2. Using a new loss function for pointer / copy mechanism. For each output token, they aggregate scores for all positions that the output token can be copied from.

I am confident that point 1 has been used in several previous works. Although point 2 seems novel, I am not convinced that it is significant enough for ICLR. I was also not sure why there is a need to copy items from the input question, since all SQL query nouns will be present in the SQL table in some form.  What will happen if we restrict the copy mechanism to only copy from SQL table.

The references need work. There are repeated entries for the same reference (one form arxiv and one from conference). Please cite the conference version if one is available, many arxiv references have conference versions.

Rebuttal Response: I am still not confident about the significance of contribution 1, so keeping the score the same.
review_title: I am not convinced that the contributions are significant

final_score: rating: 3: Clear rejection
review_content: This paper proposes a model for solving the WikiSQL dataset that was released recently.

The main issues with the paper is that its contributions are not new.

* The first claimed contribution is to use typing at decoding time (they don't say why but this helps search and learning). Restricting the type of the decoded tokens based on the programming language has already been done by the Neural Symbolic Machines of Liang et al. 2017. Then Krishnamurthy et al. expanded that in EMNLP 2017 and used typing in a grammar at decoding time. I don't really see why the authors say their approach is simpler, it is only simpler because the sub-language of sql used in wikisql makes doing this in an encoder-decoder framework very simple, but in general sql is not regular. Of course even for CFG this is possible using post-fix notation or fixed-arity pre-fix notation of the language as has been done by Guu et al. 2017 for the SCONE dataset, and more recently for CNLVR by Goldman et al., 2017.

So at least 4 papers have done that in the last year on 4 different datasets, and it is now close to being common practice so I don't really see this as a contribution.

* The authors explain that they use a novel loss function that is better than an RL based function used by Zhong et al., 2017. If I understand correctly they did not implement Zhong et al. only compared to their numbers which is a problem because it is hard to judge the role of optimization in the results.

Moreover, it seems that the problem they are trying to address is standard - they would like to use cross-entropy loss when there are multiple tokens that could be gold. the standard solution to this is to just have uniform distribution over all gold tokens and minimize the cross-entropy between the predicted distribution and the gold distribution which is uniform over all tokens. The authors re-invent this and find it works better than randomly choosing a gold token or taking the max. But again, this is something that has been done already in the context of pointer networks and other work like See  et al. 2017 for summarization and Jia et al., 2016 for semantic parsing.

* As for the good results - the data is new, so it is probable that numbers are not very fine-tuned yet so it is hard to say what is important and what not for final performance. In general I tend to agree that using RL for this task is probably unnecessary when you have the full program as supervision.
review_title: The contributions have already been done in the past

Meta-review: The pros and cons of the paper can be summarized below:

Pro:
* The improvements afforded by the method are significant over baselines, although these baselines are very preliminary baselines on a new dataset.

Con
* There is already a significant amount of work in using grammars to guide semantic parsing or code generation, as rightfully noted by the authors, and thus the approach in the paper is not extremely novel.
* Because there is no empirical comparison with these methods, the relative utility of the proposed method is not clear.

As a result, I recommend that the paper not be accepted at this time.
Decision: Reject