Venue: ICLR 2018 conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes a neural network-based approach to converting natural language questions to SQL queries. The idea is to use a small grammar to facilitate the process, together making some independence assumptions. It is evaluated on a recently introduced dataset for natural language to SQL.

Pros:
- good problem, NL2SQL is an important task given how dominant SQL is
- incorporating a grammar ("sketch") is a sensible improvement.

Cons:
- The dataset used makes very strong simplification assumptions. Not  problem per se, but it is not the most challenging SQL dataset. The ATIS corpus is NL2SQL and much more challenging and realistic:
Deborah A. Dahl, Madeleine Bates, Michael Brown, William Fisher, Kate Hunicke-Smith, David Pallett, Christine Pao, Alexander Rudnicky, and Elizabeth Shriberg. 1994. Expanding the scope of the ATIS task: the ATIS-3 corpus. In Proceedings of the workshop on Human Language Technology (HLT '94). Association for Computational Linguistics, Stroudsburg, PA, USA, 43-48. DOI: https://doi.org/10.3115/1075812.1075823

- In particular, the assumption that every token in the SQL statement is either an SQL keyword or appears in the natural language statement is rather atypical and unrealistic.

- The use of a grammar in the context of semantic parsing is not novel; see this tutorial for many pointers:
http://yoavartzi.com/tutorial/

- As far as I can tell, the set prediction is essentially predicted each element independently, without taking into account any dependencies. Nothing wrong, but also nothing novel, that is what most semantic parsing/semantic role labeling baseline approaches do. The lack of ordering among the edges, doesn't mean they are independent.

- Given the rather constrained type of questions and SQL statements, it would make sense to compare it against approaches for question answering over knowledge-bases:
https://github.com/scottyih/Slides/blob/master/QA%20Tutorial.pdf
While SQL can express much more complex queries, the ones supported by the grammar here are not very different.

- Pasupat and Liang (2015) also split the data to make sure different tables appear only in training, dev, test and they developed their dataset using crowd sourcing.

- The comparison against Dong and Lapata (2016) is not fair because their model is agnostic and thus applicable to 4 datasets while the one presented here is tailored to the dataset due the grammar/sketch used. Also, suggesting that previous methods might not generalize well sounds odd given that the method proposed seems to use much larger datasets.

- Not sure I agree that mixing the same tables across training/dev/test is more realistic. If anything, it assumes more training data and manual annotation every time a new table is added.
review_title: Re-positioning with respect to the literature is needed

final_score: rating: 7: Good paper, accept
review_content: The authors present a neural architecture for the WikiSQL task. The approach can be largely seen as graphical model tailored towards the constrained definition of SQL queries in WikiSQL. The model makes strong independence-assumptions, and only includes interactions between structures where necessary, which reduces the model complexity while alleviating the "order matters" problem. An attention mechanism over the columns is used to model the interaction between columns and the op or value in a soft differentiable manner. The results show impressive gains over the baseline, despite using a much simpler model. I appreciated the breakdown of accuracy over the various subtasks, which provides insights into where the challenges lie.
review_title: Well motivated and straightforward approach for WikiSQL

final_score: rating: 4: Ok but not good enough - rejection
review_content: This submission proposes a new seq2sel solution by adopting two new techniques, a sequence-to-set model and column attention mechanism. They show performance improve over existing studies on WikiSQL dataset.

While the paper is written clearly, the contributions of the work heavily depends on the WikiSQL dataset. It is not sure if the approach is generally applicable to other sequence-to-sql workloads. Detailed comments are listed below:

1. WikiSQL dataset contains only a small class of SQL queries, with aggregation over single table and various filtering conditions. It does not involve any complex operator in relational database system, e.g., join and groupby. Due to its simple structure, the problem of sequence-to-sql translation over WikiSQL is actually simplified as a parameter selection problem for a fixed template. This greatly limits the generalization of approaches only applicable to WikiSQL. The authors are encouraged to explore other datasets available in the literature.

2. The "order-matters" motivation is not very convincing. It is straightforward to employ a global ordering approach to rank the columns and filtering conditions based on certain rules, e.g., alphabetical order. That could ensure the orders in the SQL results are always consistent.

3. The experiments do not fully verify how the approaches bring performance improvements. In the current version, the authors only report superficial accuracy results on final outcomes, without any deep investigation into why and how their approach works. For instance, they could verify how much accuracy improvement is due to the insensitivity to order in filtering expressions.

4. They do not compare against state-of-the-art solution on column and expression selection. While their attention mechanism over the columns could bring performance improvement, they should have included experiments over existing solutions designed for similar purpose. In (Yin, et al., IJCAI 2016), for example, representations over the columns are learned to generate better column selection.

As a conclusion, I find the submission contains certain interesting ideas but lacks serious research investigations. The quality of the paper could be much enhanced, if the authors deepen their studies on this direction.
review_title: The motivation of the work is neither convincing nor verified empirically

Meta-review: The pros and cons of the paper cited by the reviewers can be summarized as follows:

Pros:
- good problem, NL2SQL is an important task given how dominant SQL is
- incorporating a grammar ("sketch") is a sensible improvement.

Cons:
- The dataset used makes very strong simplification assumptions (that every token is an SQL keyword or appears in the NL)
- The use of a grammar in the context of semantic parsing is not novel, and no empirical comparison is made against other reasonable recent baselines that do so (e.g. Rabinovich et al. 2017).

Overall, the paper seems to do some engineering for the task of generating SQL, but without an empirical comparison to other general-purpose architectures that incorporate grammars in a similar way, the results seem incomplete, and thus I cannot recommend that the paper be accepted at this time.
Decision: Reject