Venue: ICLR 2018 conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper presents a new approach to support the conversion from natural language to database queries. 

One of the major contributions of the work is the introduction of a new real-world benchmark dataset based on questions over Wikipedia. The scale of the data set is significantly larger than any existing ones. However, from the technical perspective, the reviewer feels this work has limited novelty and does not advance the research frontier by much. The detailed comments are listed below.

1) Limitation of the dataset: While the authors claim this is a general approach to support seq2sql, their dataset only covers simple queries in form of aggregate-where-select structure. Therefore, their proposed approach is actually an advanced version of template filling, which considers the expression/predicate for one of the three operators at a time, e.g., (Giordani and Moschitti, 2012).

2) Limitation of generalization: Since the design of the algorithms is purely based on their own WikiSQL dataset, the reviewer doubts if their approach could be generalized to handle more complicated SQL queries, e.g., (Li and Jagadish, 2014). The high complexity of real-world SQL stems from the challenges on the appropriate connections between tables with primary/foreign keys and recursive/nested queries. 

3) Comparisons to existing approaches: Since it is a template-based approach in nature, the author should shrink the problem scope in their abstract/introduction and compare against existing template approaches. While there are tons of semantic parsing works, which grow exponentially fast in last two years, these works are actually handling more general problems than this submission does. It thus makes sense when the performance of semantic parsing approaches on a constrained domain, such as WikiSQL, is not comparable to the proposal in this submission. However, that only proves their method is fully optimized for their own template.

As a conclusion, the reviewer believes the problem scope they solve is much smaller than their claim, which makes the submission slightly below the bar of ICLR. The authors must carefully consider how their proposed approach could be generalized to handle wider workload beyond their own WikiSQL dataset. 

PS, After reading the comments on OpenReview, the reviewer feels recent studies, e.g., (Guu et al., ACL 2017), (Mou et al, ICML 2017) and (Yin et al., IJCAI 2016), deserve more discussions in the submission because they are strongly relevant and published on peer-reviewed conferences.
review_title: This is a decent work but contains certain obvious drawbacks

final_score: rating: 4: Ok but not good enough - rejection
review_content: This work introduces a new semantic parsing dataset, which focuses on generating SQL from natural language. It also proposes a reinforcement-learning based model for this task.

First of all, I'd like to emphasize that the creation of a large scale semantic parsing dataset is fantastic, and it is a much appreciated contribution. However, I find its presentation problematic. It claims to supplant existing semantic parsing and language-to-SQL datasets, painting WikiSQL as a more challenging dataset overall. Given the massive simplifications to what is considered SQL in this dataset (no joins, no subqueries, minimal lexical grounding problem), I am reluctant to accept this claim without empirical evidence. For example, how well does the proposed model work when evaluated on an existing dataset containing full SQL queries, such as ATIS? That being said, I am sympathetic to making simplifications to a dataset for the sake of scalability, but it shouldn't be presented as representative of SQL.

On the modeling side, the role of reinforcement learning seems oddly central in the paper, even though though the added complexity is not well motivated. RL is typically needed when there are latent decisions that can affect the outcome in ways that are not known a priori. In this case, we know the reward is invariant to the ordering of the tokens in the WHERE clause. There are far simpler solutions that would achieve the same result, such as optimizing the marginal likelihood or even simply including all orderings as training examples. These should be included as baselines.

While the data contribution is great, the claims of the paper need to be revised.
review_title: Good dataset but problematic claims.

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors have addressed the problem of translating natural language queries to SQL queries. They proposed a deep neural network based solution which combines the attention based neural semantic parser and pointer networks. They also released a new dataset WikiSQL for the problem. The proposed method outperforms the existing semantic parsing baselines on WikiSQL dataset.

Pros:
1. The idea of using pointer networks for reducing search space of generated queries is interesting. Also, using extrinsic evaluation of generated queries handles the possibility of paraphrasing SQL queries.
2. A new dataset for the problem.
3. The experiments report a significant boost in the performance compared to the baseline. The ablation study is helpful for understanding the contribution of different component of the proposed method.

Cons:
1. It would have been better to see performance of the proposed method in other datasets (wherever possible). This is my main concern about the paper.
2. Extrinsic evaluation can slow down the overall training. Comparison of running times would have been helpful.
3. More details about training procedure (specifically for the RL part) would have been better.
review_title: Interesting paper, but with limited experiments

Meta-review: This paper introduces a new dataset and method for a "semantic parsing" problem of generating logical sql queries from text.  Reviews generally seemed to be very impressed by the dataset portion of the work saying "the creation of a large scale semantic parsing dataset is fantastic," but were less compelled by the modeling aspects that were introduced and by the empirical justification for the work.  In particular:

- Several reviewers pointed out that the use of RL in particularly this style felt like it was "unjustified", and that the authors should have used simpler baselines as a way of assessing the performance of the system, e.g. "There are far simpler solutions that would achieve the same result, such as optimizing the marginal likelihood or even simply including all orderings as training examples"

- The reviewers were not completely convinced that the authors' backed up their claims about the role of this dataset as a novel contribution. In particular there were questions about its structure, e.g. "dataset only covers simple queries in form of aggregate-where-select structure" and about comparisons with other smaller but similar datasets, e.g.  "how well does the proposed model work when evaluated on an existing dataset containing full SQL queries, such as ATIS"

There was an additional anonymous discussion about the work not citing previous semantic parsing datasets. The authors noted that this discussion inappropriately brought in previous private reviews. However it seems like the main reviewers issues were orthogonal to this point, and so it was not a major aspect of this decision. 
Decision: Reject