Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: In this paper, a method for re-ranking beam search results for semantic parsing is introduced and experimentally evaluated. The general idea is to train a paraphrase critic. Then, the critic is applied to the each pair (input sentence, logic form) in the beam to determine if they are close.

The main problem with the proposed method is that the critic does not receive high quality negative examples. The generator is never trained to adapt to the critic. Second big problem is that the critic trained on two sources of data: the original dataset and the Quora paraphrasing dataset. It is very unclear what is the impact of each of the data sources. Also, it is unclear how the critic works in this case. It seems to be an easy task to distinguish a logical form from a natural sentence.

In general, the paper is well written. I would suggest to reduce the size of the introduction and dedicate this space to more detailed explanation how reranking works and the experimental details. Figures don't add much to understanding.

The experimental part is rather weak. The error analysis part is great, but not very methodical. It is not clear is these examples are cherry picked or it is frequent mistake of the baseline. I would like to the accuracy of the critic and the analysis of its performance. The critic is the main contribution of this paper and it is strange that so little attention is dedicated to it. Other aspects that need to be highlighted in the experimental section: 
- how the Quora pretraining helps
- do other strategies for negative sample work
- how important is not to rerank in certain cases (Sec 3.3)

In conclusion, I encourage the authors to develop the idea further. Taking in into account the issues with the method (or its presentation) and the experimental weaknesses I recommend reject for now.

Typos:
- [CLS] is not defined in text
- Shaw et al. should be in Previous methods in Table 3
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper proposes a framework for semantic parsing, which includes a neural generator that synthesizes the logical forms from natural language utterances, and a neural reranker that re-ranks the top predictions generated by beam search decoding using the neural generator. While the neural generator is the same as prior work, the main novelty is the reranker design, which is a binary classifier that takes a pair of natural language utterance/logical form, and predicts the similarity between them. This reranker could also be pre-trained using auxiliary data sources, e.g., Quora question pairs benchmark for paraphrasing. They evaluate their approach on 3 semantic parsing datasets (GEO, ATIS, and OVERNIGHT), and show that their reranker can further improve the performance of the base generator.

I think the general motivation of the framework is sound. Although the idea of reranking is not new in the semantic parsing community, with the most recent work [1] already shows the promise of this direction, the concrete approach described in this paper is different, seems simple yet effective. The most interesting part is to transform the generated logical form into a pseudo-natural language text, so that it becomes a paraphrase of the input natural language utterance in some sense, which enables the re-ranker to be pre-trained with auxiliary data sources, and to use the wordpiece tokenizer that is effective in understanding natural language. In their evaluation, they indeed show that this transformation helps improve the performance of the reranker.

My main concern of  this paper is about evaluation. First, although they already evaluate on 3 datasets, all of them are not among the most challenging benchmarks in semantic parsing. In [1], they also evaluate on Django and Conala, which are 2 benchmarks to translate natural language to Python, and also are more complicated than the benchmarks in this paper. It would be helpful for the authors to show results on such datasets that the results of baseline neural generators are less satisfactory, which may also make more room for the possible improvement using a re-ranker.

On the other hand, they also lack a comparison with existing re-ranking approaches. For example, it will be helpful to compare with [1], given that they also evaluate on GEO and ATIS. Right now the results are not directly comparable because: (1) the base generators are different; and (2) the beam size used in this paper (10) is larger than the beam size (5) in [1]. It will be helpful if the authors can at least provide results with a smaller beam size, and would be better if they can provide results that are directly comparable to [1].

[1] Yin and Neubig, Reranking for Neural Semantic Parsing, ACL 2019.

------------
Post-rebuttal comments

I thank the authors for the response. However, I don't think my concerns are addressed; e.g., without a comparison with previous re-ranking methods, it is hard to justify their proposed approach, given that other re-ranking methods are also able to improve over an existing well-performed generator. Therefore, I keep my original assessment.
------------
review_title: Official Blind Review #2

final_score: rating: 1: Reject
review_content: This paper proposes a reranking architecture with a LogicForm-to-NaturalLanguage preprocessing step for semantic parsing. The authors experiment their method on three datasets and get the state of the art results. 

The proposed method is natural. But using neural models to rank (or rerank) is a long-existing technique, regardless of the chosen parametrization of the reranking model. This paper chose BERT. See section-2.6 of this tutorial for more details about using neural models to rank: https://www.microsoft.com/en-us/research/uploads/prod/2017/06/INR-061-Mitra-neuralir-intro.pdf. 

Overall, I think the paper is not ready to publish for the following reasons.

1. The method relies much upon manual designs that seem hard to generalize. 

By converting the logic forms to natural languages, the authors can leverage paraphrase datasets and pre-train the critic as a paraphrase model. However, the way they convert the logic forms is different for each dataset and they have to manually design rules for each logic form. 

2. It is not clear how certain experimental designs were made. 

The authors chose to not rerank if the candidates' scores are too low or high but close. Such choice and associated thresholds seem arbitrary: how were they actually found out? Were they tuned on a development set? How does the method work if the candidate with the highest score is always picked: in the end, this is what the model is supposed to learn, correct? 

Other designs include beam size, whether or not to use a pretrained model, etc. How were such decisions made? Tuned on a development set? 

3. The results are not sound enough. Given the issued pointed out in 1 and 2, I am not sure if the results are really sound as the authors claimed. 

For example, what if the authors don’t use a LogicForm-to-NaturalLanguage conversion? What is the result if we directly learn to match input and logic forms? 

Moreover, the authors better answer questions in 2 so I can gauge if their hyper-parameters were chosen in the principled ways. Once those are answered, a significant test had better be done since the improvement seems small. 

4. Claiming Shaw et al. 2019 in table-3 as ``our methods’’ is wrong. It is clear that Shaw et al. (2019) didn't experiment on OVERNIGHT dataset, but setting up the baseline on a dataset should not be classified as ``our method’’. 

Moreover, I have some comments on the model and experiments. These are not weakness, but I think some work in this direction may help improve the paper. 

1. The model architecture should be better justified. In its current form, the two arguments (input query and output sequence translated from a logic form) are interchangeable. Why so? Why isn’t an asymmetric architecture more natural? How can the authors use a pair of logic forms as negative examples (in figure-2)? Why do the authors use the Quora dataset in particular? 

2. The error analysis might be better to be a bit more quantitative. Its current form doesn’t seem to give insight on how the proposed method really helps. What the authors can do is: you can sample some sentences from the test/development set and count how many comparative words are misused in the original model, among which how many are corrected by reranking.

review_title: Official Blind Review #3

Meta-review: This paper presents and evaluates a technique for semantic parsing, and in particular proposes a model to re-rank the candidates generated by beam search. The paper was reviewed by 3 experts and received Reject, Weak Reject, and Weak Reject opinions. The reviews identified strengths of the paper but also significant concerns, mostly centered around the experimental evaluation (including choice of datasets, lack of direct comparison to baselines, need for more methodical and quantitative analysis, need for additional analysis, etc.) and some questions about the design of the technical approach. The authors submitted responses that addressed some of these concerns, but indicated that additional experimentation would be needed to address all of them. In light of these reviews, we are not able to recommend acceptance at this time, but I hope authors use the detailed, constructive feedback to improve the paper for another venue.
Decision: Reject