Improving Semantic Parsing with Neural Generator-Reranker Architecture
Authors: Huseyin A. Inan
Gaurav Singh Tomar
Huapu Pan
Venue:  (2020)
Keywords: Natural Language Processing
Semantic Parsing
Neural Reranking

Abstract: Semantic parsing is the problem of deriving machine interpretable meaning representations from natural language utterances. Neural models with encoder-decoder architectures have recently achieved substantial improvements over traditional methods. Although neural semantic parsers appear to have relatively high recall using large beam sizes, there is room for improvement with respect to one-best precision. In this work, we propose a generator-reranker architecture for semantic parsing. The generator produces a list of potential candidates and the reranker, which consists of a pre-processing step for the candidates followed by a novel critic network, reranks these candidates based on the similarity between each candidate and the input sentence. We show the advantages of this approach along with how it improves the parsing performance through extensive analysis. We experiment our model on three semantic parsing datasets (GEO, ATIS, and OVERNIGHT). The overall architecture achieves the state-of-the-art results in all three datasets. 

Under review as a conference paper at ICLR 2020
IMPROVING SEMANTIC PARSING WITH NEURAL GENERATOR-RERANKER ARCHITECTURE
Anonymous authors Paper under double-blind review
ABSTRACT
Semantic parsing is the problem of deriving machine interpretable meaning repre- sentations from natural language utterances. Neural models with encoder-decodee architectures have recently achieved substantial improvements over traditional methods. Although neural semantic parsers appear to have relatively high re- call using large beam sizes, there is room for improvement with respect to one- best precision. In this work, we propose generator-reranke architecture for semantic parsing. The generator produces list of potential candidates and the reranker, which consists of a processsss step for the candidates followed by novel critic network, reranks these candidates based the similarity between each candidate and the input sentence. We show the advantages of this approach along with how it improves the parsing performance through extensive analysis. We experiment our model on three semantic parsing datasets (GEO, ATIS, and OVERNIGHT). The overall architecture achieves the state-of-the-ar results in all three datasets.
INTRODUCTION
Semantic parsing is the task of deriving machine interpretable meaning representations such as logi- cal forms structured queries from natural language utterances. These meaning representations can be executed in various environments, making semantic parsing applicable in many frameworks such as querying data/knowledge bases for question answering (Zelle & Mooney 1996 Zettlemoyer & Liang et al. 2011 Berant et al.) (013), generating regular expression (Kushman & |arzzlay 2013), instruction following (Artzi Zettlemoyer 2013), and communicating with robots
Collins
2007
(Chen & Mooney 2011 Tellex et al. 201 Bisk et 2016).
Conventionally, semantic parsing has been done with a two step approach: first, a large number of potential candidates are generated using deterministic methods and combinatorial search (the gen- erator), and then the best candidate is selected among them with a probabilistic method or scoring (the critic) (Berant & Liang 2014 Kwiatkowski al. 2013 Zettlemoyer & Collins 2005 Berant cct al.) 2013 Cai & Yates] 2013). With recent advancement of neural networks, neural models with encoder-decoder architectures has obtained impressive improvements (Dong & Lapata 2016 Jia & Liang 2016) Herzig & Berant, 2017 Su & Yan [017 B. Chen & Han, 2018 (Shaw et al.) 2019). These encoder-decoder based neural semantic parsers produce one candidate given an input sen- tence, essentially acting as both the generator and the critic. However, undesirable prediction errors still occur, e.g. predicting wrong comparative or superlative structure (such < instead of <). On the other hand, neural semantic parsing models have high recall, i.e. top-n predictions of the model cover the gold-stan standdt meaning representation most of the time (c.f. Table Section
In this work we propose generator-rerankee architecture that uses two neural networks for semantic parsing: generator network, which generates a list of potential candidates, and reranker system, which consists of a pre-processing step for the candidates followed by novel critic network that reranks these candidates based on the similarity between each candidate and the input sentence. An advantage of separating the semantic parsing process into generator network and a critic network that the critic observes each candidate and the input sentence entirely, taking into account bidi- rectional representations of both sentences and globally reason over the entire candidate. This may be more effective in terms of choosing the right candidate and mitigating some the errors arising from auto-regressive decoding in the generator. Furthermore, the critic leverage extra

---

Under review as a conference paper at ICLR 2020
Table 1: One example from each dataset that in experiments. Input utterances and the corresponding logical forms are denoted by æ and respectively.
Dataset
Example
GEO
"which states adjoin alabama ?"
y answer (state (next to (stateid (atateidalaaaan )
ATIS
v "gel flights between petersburg and charlotte"
y (.lambda $0 e (Land (-flight $0)
(-from S0 st.petersburg:.cii (_to $0 charlotte:.ci))
OVERNIGHT
x "show me all meetings not ending at 10 am
y Type Meeting
EndTime != 10
data sources for training, e.g. a paraphrase dataset and allow for better transfer learning. Our key contributions this work are the following:
We introduce neural critic model, which reranks the candidates of semantic parser based on their similarity score with respect to the input utterance.
2. We propose various prepprocessing methods for the candidates to leverage the pre-trained repre- sentations of their tokens using the critic model.
We show through extensive qualitative and quantitative studies how the critic model helps miti- gating errors of a state-of-the-art neural semantic parser and improves the performance.
Evaluation results of our approach on three existing semantic parsing datasets (see Table for sample input-output pair for each dataset) show that our model improves upon the tfte-of-the-art results and the renarar architecture can substantially improve parsing performance.
RELATED WORK
The semantic parsing problem has received significant attention and has rich literature (Kamath &
Das 2018), While traditional approaches (Kate & Mooney| [2066] Wong & Mooney] 2007/Cllrrk Clarke
al. 2010, Zettlemoyer & Collins, 2007: Kwiatkowski al. [011] Wang al. 2015
al.
2015
Cai & Yates 2013
(0133/Bere Berant et al. [201]|uurk Quirk et al. [215]] Artzi et al. 2015] Zhang et al. 2017
rely on high-quality lexicons, manually-built templates, and/or domain or representation specific features, in more recent studies neural models with encoder-decoder architectures show impressing
results (Dong & Lapata 2016 Jia & Liang 2016
Herzig Berant 2017
Yan 2017
Chen
& Han 2018
Shaw et al. 2019))
Among recent results, Herzig & Berant (017) achieves the best performance on the OVERNIGHT dataset WWagg et al. 2015), which consists of 8 various domains. The model in Herzig & Berant (2117) is an attention basee
model and it is trained jointly over 8 domains. Very recently, (Shaw et al. (2019) presents approach that uses Graph Neural Network (GNN) architecture, which successfully incorporates information about relevant entities and their relations in parsing natural utterances. Similar Vinyals al. (2015): Dia & Liang (0166); |Hrzii & Berant (007)). the decoder has copying mechanism, which can copy an entity to the output during parsing. This model achieves the best performance GEO dataset (Zelle & Mooney (996) and competitive
with state-of-the aft in ATIS dataset (Dahl et al. 1994). We note that this model was not applied to the OVERNIGHT dataset.
We provide in Table the comparison between top-10 10 (25) oracld" and top-1 1 (greedy decoding) accuracy for the state-of-the-an sequence-to-sete to-seqennc models. We that while may not be possible to reach t0p-10 (25) accuracy for all dataset there is certainly room for improvement.
'The fraction test examples where the gold-standard logical form present in one the top-10 (25) predicted logical forms. For instance, Wang al. 2015) reports 17% of incorrect labels in the OVERN GHT dataset.
2

---

Under review as a conference paper at ICLR 2020
Top-1 (greedy decoding) top-10 (25) oracle for the best performing sequence-to sequence models on three semantic parsing datasets.
Table 2:
Model
Dataset
Top-1 1 Acc.
Top-10 Oracle
Top-25 Oracle
Herzig & Berant (017)
OVERNIGHT
79.6
91.7
93.5
Shaw et al.
2019)
GEC
92.5
95.3
96.4
Shaw et al.
2019
A2IS
89.7
93.3
94.2
Mpput otterange
Reranker
Input utterance
Generator Shaw et 2019)
Process candidates
Critic (BERT)
Logical form
logical
form Logic form
Processed
Proneesed Ingical form Processed logical form
Logital form
Log form
Figure I: Overview of our method demonstrating semantic parsing process. In the example, the top prediction by the generator is whereas the candidate y2 is scored highest by the critic among the candidates of the generator.
Motivated by these observations, we consider generator-reranker architecture for semantic parsing. The reranker system consists of critic network, which reranks candidate logical forms based on their similarity scores to the input utterance. We propose various techniques for processing candidate logical forms before the critic scores their similarity to the input utterance. For the generator, we use the model in Shaw et al. (2019) and for the critic, we use the BERT model (Devlin et al.) (018) for all three datasets. Although the setting is parser agnostic, we would like to investigate the reranke architecture can further improve the performance of the best performing parser.
To the best of our knowledge, our reranker system novel and has not been considered before in semantic parsing. However, the idea of reranking the candidates of generator model appears in various applications in the literature. We refer the reader to Collins Koo (2005) for reranking based on combination of feature functions for various NLP tasks.
For semantic parsing task, Yavuz et al. (2016) applies reranking based on predicted answer type. We note that this method may be effective for the cases where the candidate logical forms return the same answer type but the top prediction is wrong (such as predicting wrong comparative, e.g. instead of <). We believe choosing the best candidate by assessing its relevance to the input utterance entirely would be more effective method. In more recent work by Yin & Neubig 2019), measuring features of candidate logical forms. The first
reranking is applied by two main one is the reconstruction feature, using the probability of reproducing the original input utterance from y. The second is the discriminative matching feature, which is based on pair-wise associations of tokens in æ and y. In our work, reranking of candidate logical forms is applied based on their similarity with the input utterance directly using a critic model which can leverage the pre-trained representations of processed logical forms as well as extra data sources to learn similarity.
MODEL ARCHITECTURE
Our goal is learn a model which maps an input utterance æ logical form representation of its meaning y. The input ufterance is sequence of words 21, .2,...., yiin) where (in) is the input vocabulary and the output logical form y is sequence of tokens y1, V2,-..; E (ouut) where (out) is the output vocabulary.
In this work, we consider a combination of two models for semantic parsing. Figure T Dillustrates the model architecture along with example. We describe the model architecture in what follows.
3

---

Under review as a conference paper at ICLR 2020
3.1 GENERATOR
To generate candidate logical forms, we use the model recently introduced in Shaw et al. (2019). which is based on the Transformer architecture (Vaswani al. 2017), with the self-attention layer extended to incorporate relations between imput elements, and the decoder extended with copy mechanism similar to Vinyals al. (2015); Dia & Liang (); Herzig & Berant (2017). The model is trained with natural utterances paired with logical forms and learns shared representations from these pairs. Candidate logical forms are generated using beam search for given input utterance.
3.2 PROCESSING CANDIDATES
Before scoring the similarity between logical form and input utterance, we preprocess the didate logical forms. We propose the processing methods in an increasing order of their complexity and how close they map the logical forms natural text. Note that the critic can leverage the pretrained representations of more processed logical forms better and can be more effective when scoring similarities. Figure Appendix illlustrates these methods along with an example.
3.2.1 RAW LOGICAL FORM
In this method, consider raw logical forms without any processing for calculating similarity with respect to the input utterance. This is the simplest method that is applicable to any dataset.
3.2.2 NATURAL LANGUAGE ENTITY NAMES
In this method, we simply convert entities to natural text. The output tokens are often self- explanatory and easy be converted simple text, e.g., "num.assists" "number of assists", "em.location.greeneee www.ngrreeeeeeooe _cafe" + "greenberg cafe" in OVERNIGHT dataset, and "_arrival "arrival time" in ATIS dataset etc. This approach requires an additional step but can be applied in straightforward manner. The advantage of this approach is that the critic can leverage pre-trained representations for these tokens. We describe the exact procedure Appendix A for each dataset.
3.2.3 TEMPLATED EXPANSIONS
In this method, logical forms are converted canonical utterances using a deterministic template (e.g. arg max (type. player, numRebounds) to "player that has the largest number of re- bounds"). The purpose of the canonical utterances is to capture the meaning of the logical forms. Here the assumption is that while it is nearly impossible to generate a grammar that parses all utter- ances, it is possible to write one that generates one canonical utterance for each logical form (Wang et al. 2015). We use this method only for OVERNIGHT dataset as there is an available template in- troduced in Wang et al. (2015) to convert the logical forms to canonical utterances. In this case, the critic can be considered
paraphrase model, producing score based the equivalence of two sentences semantically. During test time, candidate logical forms that do not have a corresponding canonical utterance are treated as incorrect candidates and they not included in ranking.
3.3 CRITIC
The critic network takes two sentences as an input and outputs a score E [0, 1] based on the similarity of the sentences. The input may consist of two arbitrary sentences (coming from different language models, vocabularies etc.) depending on the processing of the logical forms described in the previous section. We use the critic to rerank the candidate logical forms based on their similarity score with respect the input utterance. We use the BERT model (Devlin al. 2018) for this tas ]
To train the critic, positive and negative examples are generated as follows. Each natural utterance in the training set has stad-standard logical form, which forms the positive examples. For the negative examples we apply the generator, which is trained over the training set, to generate logical forms for each natural utterance in the training set with beam search. The incorrect logical forms paired with the natural utterance for negative examples. Additionally, we pair any two logical forms
'hhe logical forms are tokenized using the public BERT wordpiece tokenizer as wordpieces often contain useful information. Different tokenization methods considered, which we leave for future work.
4

---

Under review as a conference paper at ICLR 2020
Processed logical form
Training example
Generator Shaw et al. 2019)
Process candidates
Procossed logical form
Procimserd logicall form
y2 Logical
Positive pain: (4. S.) Negative 8c
input fori
yy Logical
yg Logical
Figure Overview of generating training examples for the critic.
among beam candidates as negative examples to increase training data and help the model learn subtle differences among the candidates. The logical forms are processed according to the method use in the model. The positive pairs are labeled as 1 and the negative pairs labeled as O. The model performs binary classification. Figure Saaure@illin iillustrates this approach with an example.
As aforementioned, we can leverage existing paraphrase datasets pretrain the critic and fine tune it over the examples we generate. For instance, we use the Quora question pairs" which contains over 400K annotated question pairs with binary paraphrase labels. Furthermore, we may choose not to apply reranking in certain cases. From the error cases of the critic over the training examples, two cases stand out. The first one is where each candidate is scored below 0.5 (no candidate similar to the input utterance according to the critic) and the second one is the case where at least two candidates are scored very high and close to each other. It appears quite natural not to do ranking all scores are below 0.5 to address the first case. On the other hand, one can address the second case by setting a threshold between the best and second best score and choose not to do ranking the difference is less than this threshold. However, setting this threshold is prone overfitting and may look somewhat arbitrary if chosen by maximizing the accuracy over the evaluation set. Therefore, we set this threshold as 0.001 once and keep same for all experiments. Our goal here is merely to show various strategies one can take with the critic model. We may choose not to rerank if cither or both of these cases occur inference and instead output the top prediction of the generator.
EXPERIMENTS
4.1 DATASET
We use three semantic parsing datasets in our experiments. Table [ presents an example for each dataset.
GeoQuery (GEO) contains natural language questions about US geography along with correspond- ing logical forms (Zelle & Mooney 1996). We follow Zettlemoyer & Collins (2005) and use 600 training examples and 280 test examples. We use logical forms based on Functional Query Language (FunQL) Kate et al. 2005). We use logical form exact match when reporting accuracy.
ATIS (ATIS) contains natural language queries for a flights database along with corresponding database queries written in lambda calculus (Dahl al. 1994). We follow Zettlemoyer & Collins (207) and use 4473 training examples and 448 test examples. We compare normalized logical forms using canonical variable naming and sorting for unordered arguments (Jia & Liang 2016 when reporing accuracy.
Overnight (OVERNIGHT) contains 13,682 examples of language utterances paired with logical forms across eight domains (Wang et al. 2015). In this dataset, each logical form has corre- sponding canonical utterance, which we use in templated expansions method when we process the candidate logical forms canonical utterances. We evaluate on the same train/test split Wang et al. (2015): Jia & Liang (2016); Herzig & Berant (2077) with the same accuracy metric, i.e. the fraction of test examples for which the denotations of the predicted and gold logical forms are equal.
4.2 TRAINING DETAILS
For the generator, we follow the settings of Shaw et al. (011 for GEO and ATIS datasets. The model was not applied OVERNIGHT dataset, therefore, we configured the hyperparameters based
See
https::/datta.uuora.coo quora www.://.............../ ott.aaa......aaaaaaooaooooaoaaaa Daraaaaaataseeee aset Release- Quest innestion-Pae
5

---

Under review as a conference paper at ICLR 2020
Test accuracy for all models OVERNIGHT dataset, which has eight domains: Basketball, Blocks, Calendar, Housing, Publications, Recipes, Restaurants, and Social. We use the generator- reranker (G-R) architecture with different options. Beam-n: Beam search is applied with size n. pQ: The critic is pre-trained over the Quora dataset, THI: rerank if there is at least one score above 0.5, TH2: rerank best score second best score > 0.001. The candidate logical forms are processed with templated expansions method (Section 3.2.3 in this experiment.
Table 3:
Method
Bas.
Blo.
Cal.
Hou.
Pub.
Rec.
Res.
Soc.
Avg.
Previous Methods
B. Chen & Han 20188)
88.2
61.4
81.5
74.1
80.7
82.9
80.7
82.1
79.0
Su & Yan 2017
88.2
62.2
82.1
78.8
80.1
86.1
83.7
83.1
80.6
Herzig & Berant (2017)
86.2
62.7
82.1
78.3
80.7
82.9
82.2
81.7
79.6
Our Methods
Shaw et al. (2019)
89.3
63.7
81.5
82.0
80.7
85.6
89.5
84.8
82.1
G-R (Beam-10)
88.7
66.4
83.3
82.5
78.9
86.6
89.8
83.7
82.5
G-R (Beam- 10 & pQ)
89.0
65.2
83.3
83.6
78.3
87.5
89.5
85.5
82.7
G-R (Beam-23
89.0
67.7
83.3
84.1
82.6
87.5
89.4
83.9
83.4
G-R (Beam-25 & pQ)
89.3
66.7
84.5
83.6
80.1
88.0
91.0
85.2
83.5
G-R (Beam-25 pQ TH1)
89.0
65.7
85.1
83.6
81.4
88.0
91.0
86.0
83.7
G-R (Bcam-25& & pQ & TH2)
88.7
66.4
82.7
83.1
82.0
87.0
89.8
85.8
83.2
on performance cross-validated on the training set. We provide the final setting and training process in full detail in Appendix B
For the critic, use the BERTLARGE model in Devlin al. (2018), where we feed two sentences as an input and feed the [CLS] representation into an output layer for binary classification. We either directly train the model with the examples generated by the generator (see Section (.3) or train the model on Quora question pairs first and fine tune it over our examples. We apply the early stopping rule based on the evaluation set accuracy to determine the total training steps. We use the learning rate 1e-6 and batch size 32 when we train and fine tune the critic.
4.3 RESULTS
OVERNIGHT: We compare our model with the state-of-the-ar models Table 3 All models use the same training/test splits, therefore, directly take the reported best performances from their original papers for fair comparison. We use the model in Shaw et al. (2019) for the generator. As Table 3 shows, this model alone (without any reranking) improves the state-of-the-an performance ff-the-art art In this experiment, we use templated expansions method (Section 3.2.3) when processing candidate logical forms for reranking. We do our experiments over the generator-reranker architecture on three settings:
from 79.6% to 82.15% accuracy and sets a new state-of-th faaee-of-the-ad
sequence- to-eeueee---eeeence model.
1. Beam size This determines the number of candidates produced by the generator.
2. Initialization of the critic This determines how the critic is initialized. We pre-trained model (pQ), which trained over Quora question pairs and fine tuned over the generated ex- amples by the generator. We compare this with directly training the critic over the generated examples by the generator without Quora dataset.
Reranking with a threshold rule We apply reranking based on various threshold rules. We compare reranking at all times with reranking when there is at least one candidate with score above 0.5 (THI) and reranking if the difference between best score and second best score is above 0.001 (TH2).
From the overall results, we can see that:
'n Su & Yan 2017). each domain has separate model where it is trained using the training data of all the other domains and fine tuned on the training data of the domain reported. All the other methods have one single model.

---

Under review as a conference paper at ICLR 2020
1. Increasing the beam size improves the performance expected. As the generator outputs more candidates, it is more likely that the correct form is among them. Therefore, this allows the critic to identify a higher number of true positives and improve the performance. Increasing the beam size further does not significantly improve the performance (beam size 50 achieves 83.80% accuracy), hence we conclude the experiment with beam size 25.
2. Using a pre-trained model improves the performance as well. We note that the task of the critic to infer the similarity of two input sentences, therefore, we can initialize it with one that has been trained over an existing paraphrase dataset.
3. Reranking with threshold rule may be helpful for the overall architecture. We observe that reranking by the critic at all times may not be the best approach. We note that choosing not to rerank when all scores are below 0.5 increases the performance further. On the other hand, reranking if the difference between the best score and second best score is above the threshold we set does not help in this case.
The overall architecture improves the performance of the generator (82.1% accuracy) to 83.7% accuracy. We note that this improvement is significant for OVERNIGHT dataset as it is an average over 8 domains with an improvement for each one of them.
We next apply raw logical form (Section 3.2.1 and natural language entity names (Section 3 3.2.2
methods when processing the candidate logical forms and show that the critic improves the per- formance in these cases as well. While best result with templated expansions (Section 3.2.3)
method achieves 83.71% 1% accuracy in Table the best results we achieve with the first two methods 82.81 % and 83.16% respectively. We note that the critic improves upon the performance of the generator (82. % accuracy) in all three cases. As the logical forms are processed more towards natural text. the performance gets better. This is expected since helps the critic measure the similarity with respect to the input utterance.
GEO and ATIS: We continue with GEO and ATTS datasets and provide the best set of our results.
For the GEO dataset, we set the beam size as 10 for the number of candidates produced by the generator. We do not process the candidate logical forms and use the raw versions as the output tokens are already in processed form. We pretrain the critic over the Quora dataset and fine tune it over the generated examples by the generator on the training set. We produce 25 candidate logical forms for each example in the training set when generating examples to train the critic.
For the ATIS dataset, we set the beam size as 10 for the number of candidates produced by the generator. We process the candidate logical forms with natural language entity names method in straightforward manner (see Appendix (.TT). We pretrain the critic over the Quora dataset and fine tune it over the generated examples by the generator on the training set. We produce 25 candidate logical forms for each example in the training set when generating examples to train the critic.
We observe performance gain in both datasets and achieve the state-of-the-arr performance with the generator-reranker architecture, We note that in ATIS dataset, there is significant improvement upon the bascline and the overall architecture obtains the state- of-the-art result over Wang et al. (2014), where the approach is not based on neural models.
The results are shown in Table 4
4.4 ERROR ANALYSIS
In this section, we categorize the types of errors the generator model makes and analyze which ones are corrected by the critic to understand what types of errors can be mitigated via our approach. We use the OVERNIGHT dataset as it has a much larger test set. In the following examples, the top prediction of the generator is wrong, but the gold-standard logical form is among the beam candi- dates and correctly scored highest by the critic. We provide the corresponding canonical utterances instead of the logical forms for the sake of presentation.
1. Wrong comparative structure:
Input utterance: meetings after january 2 or after january 3 Top prediction: meeting whose date is at least jan 2 or jan
Corrected to: meeting whose date is larger than jan 2 or jan 3
7

---

Under review as a conference paper at ICLR 2020
Table 4: Test accuracy for all models GFO and ATTS datasets. The settings follow the Table 3 and we denote TH3 as the threshold rule of applying both THI and TH2, i.c. reranking
when there least one score above 0.5 and if best score second best score > 0.001.
Method
ATIS
Method
GEO
Previous Methods
Previous Methods
Wang et al.
(2014)
91.3
Shaw et al. (2019)
92.5
Shaw et al.
2019
89.7
Our Methods
Our Methods
G-R (Beam- 10 pQ)
92.5
G-R (Beam-10 & pQ)
90.6
G-R (Beam- 10 & pQ & THI)
92.5
G-R (Beam-10 & pQ & THI)
91.1
G-R (Beam-10 10 pQ TH2)
93.2
G-R (Beam-10 & pQ & TH2)
91.3
G-R (Beam-10 & pQ & TH3)
93.2
G-R (Beam-10 pQ & TH3)
91.5
NP-shift to wrong position:
Input utterance: what article citing multivariate data analysis was in annals of statistics Top prediction: article whose venue is annals of statistics and that multivariate data analysis cites Corrected to: article whose venue is annals of statistics and that cites multivariate data analysis
Similar word confusion in elliptical constructions:
Input utterance: show me recipes with preparation time equal to greater than rice pudding Top prediction: recipe whose preparation time is at least cooking time of rice pudding
Corrected to: recipe whose preparation time is least preparation time of rice pudding
4. Incorrect matching to semantically non-equivalent phrase: Input utterance: housing that is cheaper than 123 sesame street
Top prediction: housing unit whose size is smaller than size of 123 sesame street Corrected to: housing unit whose monthly rent smaller than monthly rent of 123 sesame street
Although the critic helps fixing these errors and improves the performance in semantic parsing, there are few cases where it is ineffective. Therefore, stronger critic may further increase the performance. We provide here the cases where the top prediction of the generator is correct, but the critic scores another prediction higher and this leads to an error.
1. Comparative by trivial algebra:
Input utterance: friends of people who joined their jobs before 2005
Top prediction: person that employee whose start date is at most 2004 friends with Best scored candidate: person that employee whose start date is smaller than 2004 is friends with
Comparative by common sense: Input utterance: who is younger than alice
Top prediction: person whose birthdate is larger than birthdate of alice
Best scored candidate: person whose birthdate is smaller than birthdate of alice
5 CONCLUSION
In this paper, we proposed a generator-reranker architecture for semantic parsing. We introduced novel neural critic model that reranks the candidates of a semantic parser based their similarity scores with respect to the input utterance. We proposed various processing methods for the candidate logical forms, enabling the critic to leverage the pre-trained representations for the tokens of the candidates. Our architecture further enables leveraging extra data resources in a direct fashion. We showed that the proposed architecture improves the parsing performance and achieves the state-of- the-art results three semantic parsing datasets.
As the model generator agnostic, in the future work we plan to try (a combination of) neural and/or traditional parsers as the generator and apply our architecture on more benchmark datasets. In addition, we believe that our architecture may also be effective the cross-lingual semantic parsing setting (Zhang al. 2018). One leverage paraphrase datasets available in source- target language and pretrain the critic, which could help choosing the right form among the generator candidates.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Yoav Artzi and Luke Zettlemoyer. Weakly supervised learning of semantic parsers for mapping instructions to actions. Transactions of the Association for Computational Linguistics, 1:49-62, 2013.
Yoav Artzi, Kenton Lee, and Luke Zettlemoyer. Broad-coverage CCG semantic parsing with AMR. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 1699-1710, September 2015.
L. Sun B. Chen and X. Han. Sequence-to-actione End-to-end semantic graph generation for semantic parsing. Proceedings of the Annual Meeting of the Association for Computational Linguistics, pp. 766-777,20 2018.
Jonathan Berant and Percy Liang. Semantic parsing via paraphrasing. Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp- 1415-1425, June 2014.
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on Freebase from question-answe answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1533-1544, 2013.
Yonatan Bisk, Deniz Yuret, and Daniel Marcu. Natural language communication with robots. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Com- putational Linguistics, pp. 751-761. Association for Computational Linguistics, 2016.
Qingqing Cai and Alexander Yates. Large-scall scale semantic parsing via schema matching and lexicon extension. In Proceedings of the Annual Meeting of the Association for Computational Linguis- tics, 2013.
David L. Chen and Raymond J. Mooney, Learning to interpret natural language navigation instruc- tions from observations. In Proceedings of the Twenty-Fifit National Conference on Artificial Intelligence, AAAL, pp. 859-865- 2011.
James Clarke, Dan Goldwasser, Ming- Wei Chang, and Dan Roth. Driving semantic parsing from the world's response. In Proceedings of the Fourteenth Conference Computational Natural Language Learning, pp. 18-27, July 2010.
Michael Collins and Terry Koo. Diseriminative reranking for natural language parsing. Comput. Lingwist., 31(1):25-70, March 2005.
Deborah A. Dahl, Madeleine Bates, Michael Brown, William Fisher, Kate Hunicke-Smith. David Pallett, Christine Pao, Alexander Rudnicky, and Elizabeth Shriberg, Expanding the scope of the ATIS task: The ATIS-3 corpus. In Proceedings the Workshop on Human Language Technology, pp. 43 48, 1994.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. ArXiv, abs/1810.04800 2018.
Li Dong and Mirella Lapata. Language logical form with neural attention. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pp. 33-43, 2016.
Herzig and J. Berant. Neural semantic parsing over multiple knowledge-bases. Association for Computational Linguistics (ACL), 2017.
Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 328-332-33 228-339, July 2018
Robin Jia and Percy Liang. Data recombination for neural semantic parsing. In Proceedings of the 54th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 12-22, August 2016.

---

Under review as a conference paper at ICLR 2020
Melvin Johnson, Mike Schuster, Quoe V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's s multilinguul neural machine translation system: Enabling zero-shot translation. Trans- actions of the Association for Computational Linguistics, 5:339-351, 2017.
Aishwarya Kamath and Rajarshi Das. A survey on semantic parsing. CoRR, abs/ 1 1812.00978, 2018.
Rohit J. Kate and Raymond J. Mooney. Using string-kernlss for learning semantic parsers. In Proceedings of the 21st International Conference on Computational Linguistics and the 44th Annual Meeting of the Association for Computational Linguistics, pp-
2006.
Rohit Kate, Yuk Wah Wong, and Raymond J. Mooney. Learning to transform natural formal languages. In Proceedings of the 20th National Conference on Artificial Intelligence Volume 3, pp. 1062-1068, 2005.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference Learning Representations, December 2014.
Nate Kushman and Regina Barzilay. Using semantic unification generate regular expressions from natural language. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics, pp. 826 836, 2013.
Tom Kwiatkowski, Luke Zettlemoyer, Sharon Goldwater, and Mark Steedman. Lexical generaliza- tion in CCG grammar induction for semantic parsing. In Proceedings of the 201 Conference on Empirical Methods in Natural Language Processing, pp. 1512-1523, July 2011.
Tom Kwiatkowski, E Choi, Y Artzi, and L Zettlemoyer. Scaling semantic parsers with on-the-fly ontology matching. EMNLP 2013 2013 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference, pp. 1545-156,, January 2013.
Junhui Li, Muhua Zhu, Wei Lu, and Guodong Zhou. Improving semantic parsing with enriched synchronous context-free grammar. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 1455-1465, September 2015.
Percy Liang, Michael I. Jordan, and Dan Klein. Learning dependency-baseee baseed compositional seman- tics. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics, pp. 590-599, 2011.
Chris Quirk, Raymond Mooney, and Michel Galley. Language to code:
Learning semantic parsers for if-this-then-that recipes. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Pro- cessing (Volume 1: Long Papers), pp. 878-888, July 2015.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representa- tions. In Proceedings of the 2018 Conference of the North American Chapter the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pp. 464 468, June 2018
Peter Shaw, Philip Massey, Angelica Chen, Francesco Piccinno, and Yasemin Altun. Gener- ating Logical Forms from Graph Representations of Text and Entities. arXiv e-prints, pp- 10X.v.1905.08407, May 2019.
Yu Su and Xifeng Yan. Cross-domain semantic parsing via paraphrasing. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 1235-1246, 2017.
Stefanie Tellex, Thomas Kollar, Steven Dickerson, Matthew R. Walter, Ashis Gopal Banerjee, Seth Teller, and Nicholas Roy. Understanding natural language commands for robotic navigation and mobile manipulation. In Proceedings of the Twenty-Fifth National Conference on Artificial Intel- ligence, AAAI, pp. 1507-1514, 2011.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems 30, pp. 5998-6008. 2017.
10

---

Under review as a conference paper at ICLR 2020
Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. Pointer networks. In Advances in Neural Information Processing Systems 28, pp. 2692-2700. 2015.
Adrienne Wang, Tom Kwiatkowski, and Luke Zettlemoyer. Morpho-syntactic lexical generalization for CCG semantic parsing. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1284-1295, October 2014.
Yushi Wang, Jonathan Berant, and Percy Liang. Building semantic parser overnight. In Proceed- ings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference Natural Language Processing, pp. 1332-13422 2015.
Yuk Wah Wong and Raymond Mooney. Learning synchronous grammars for semantic parsing with lambda calculus. In Proceedings of the 45th Annual Meeting of the Association of Computational Lingnistics, pp. 960-967, June 2007.
Semih Yavuz, Izzeddin Gur, Yu Su, Mudhakar Srivatsa, and Xifeng Yan. Improving semantic pars- ing via answer type inference. In Proceedings of the 2016 Conference Empirical Methods in Natural Language Processing, pp. 149-159, November 2016.
Pengcheng Yin and Graham Neubig. Reranking for neural semantic parsing. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4553-4559, July 2019.
John M. Zelle and Raymond J. Mooney. Learning to parse database queries using inductive logic programming. In Proceedings of the Thirteenth National Conference on Artificial Intelligence, AAAI, pp. 1050-1055, 1996.
Luke Zettlemoyer and Michael Collins. Online learning of relaxed CCG grammars for parsing to logical form. In Proceedings of the 2007 Joint Conference Empirical Methods in Natu- ral Language Processing and Computational Natural Language Learning TEMNLP--CNNLLL pp- 678-687, 2007.
Luke S. Zettlemoyer and Michael Collins. Learning to map sentences to logical form: Structured classification with probabilistic categorial grammars. Proceedings of the Twenty-Firs. Confer- ence on Uncertainty in Artificial Intelligence, pp. 658-666, 2005.
Sheng Zhang, Kevin Duh, and Benjamin Van Durme.
Cross-lingual semantic parsing. CoRR,
abs/1 804008037.. 2018
Yuchen Zhang, Panupong Pasupat, and Percy Liang. Macro grammars and holistic triggering for efficient semantic parsing. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 1214-1223, September 2017.
A PROCESSING CANDIDATES
A.1 NATURAL LANGUAGE ENTITY NAMES
We apply this method to OVERNIGHT and ATIS datasets as the CEO dataset does not require pro- cessing since the output tokens already has English words.
For the ATIS dataset, we apply a very simple procedure. We take the tokens that start with an underscore (i.e. "_") and remove (e-g. "_from" - "from", "_to" - "to" etc.). Additionally, for such tokens, if there are underscores in between, replace them with space break into further tokens (c.g. " "_departure_time"
"departure time". " "_fare_amoun + "fare amount").
For the OVERNIGHT dataset, each entity token has corresponding English text available in Wang et al. (2015). which used in the process of building the dataset. These are used within a set of rules for the logical form, canonical utterance conversion well. We note that this method is rather straightforward compared to templated expansions method since we only replace each entity token with its corresponding English text.

---

Under review as a conference paper at ICLR 2020
x: restourants that offer takeout and are price rated 2 doller sgge
y: typeRestaurant
HasTakeeu PriceRating. 25:
Process candidates
Process candidates
Process candidates
(Raw Logical Form)
(Natural Language
(Templated Expansions)
Entity Names)
Type Restauratt (HasTakeout, PriceRating... 2
resta urant take-ouch price rating 2
restaurent whose price rating is signs and
Using logical forms direcily
Converling
Converting logical canonical ullerancee
Figure 3: Overview of the proposed three methods with example demonstrating each Raw logical form method uses logical forms as they are without any change, natural language entity names method simply changes entities natural text, and templated expansions method uses template to convert logical forms to canonical utterances.
B TRAINING DETAILS OF THE GENERATOR ON OVERNIGHT DATASET
The final setting has 2 encoder and 2 decoder layers. The source embedding and hidden dimensions are chosen as 64 and the target embedding and hidden dimensions are chosen as 128. We set the corresponding feed forward layer hidden dimensions 4 times higher. We employed dropout at train- ing time with probability 0.2. We used 8 attention heads for each task. We used a clipping distance of 8 for relative position representations (Shaw et al. 2018).
We used the Adam optimizer (Kingma Ba 2014) with 31 0.9, B2 = 0.98, and 1e-9. We set the learning rate 5e-5. We used the same warmup and decay strategy for learning rate as Vaswani jtt al. (2017), selecting the warmup steps as 4000. We trained the model for 50000 steps with batch size 32. We used simple strategy of splitting each input utterance on spaces to generate a sequence of tokens. We mapped any token that did not occur at least times in the training dataset to a special out-of-vocabulart token. Token embeddings are taken from pre- traitrained BERT (Devlin al. 2018) encoder. We freeze the pre-trained parameters of BERT for 6000 steps, and then jointly fine tune all parameters, similar existing approaches for gradual unfreezing (Howard & Ruder 2018). When unfreezing the pre-trained parameters, we restart the learning rate schedule.
Motivated by the best performing approach in Herzig & Berant (0017). the generator is trained ing the natural utterance, logical form pairs over all training examples, i.e., jointly over all domains. Therefore, all model parameters are shared across domains and the model trained from all exam- plcs. Since there is no explicit representation of the domain that is being decoded, to help the model learn identifying the domain given only the input, we add an artificial token at the beginning each source sentence to specify the target domain, similar to Johnson et al. (2017) for neural machine translation.
12


Reviews:
[{"review_id": "HygYH9sl5H", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, a method for re-ranking beam search results for semantic parsing is introduced and experimentally evaluated. The general idea is to train a paraphrase critic. Then, the critic is applied to the each pair (input sentence, logic form) in the beam to determine if they are close.\n\nThe main problem with the proposed method is that the critic does not receive high quality negative examples. The generator is never trained to adapt to the critic. Second big problem is that the critic trained on two sources of data: the original dataset and the Quora paraphrasing dataset. It is very unclear what is the impact of each of the data sources. Also, it is unclear how the critic works in this case. It seems to be an easy task to distinguish a logical form from a natural sentence.\n\nIn general, the paper is well written. I would suggest to reduce the size of the introduction and dedicate this space to more detailed explanation how reranking works and the experimental details. Figures don't add much to understanding.\n\nThe experimental part is rather weak. The error analysis part is great, but not very methodical. It is not clear is these examples are cherry picked or it is frequent mistake of the baseline. I would like to the accuracy of the critic and the analysis of its performance. The critic is the main contribution of this paper and it is strange that so little attention is dedicated to it. Other aspects that need to be highlighted in the experimental section: \n- how the Quora pretraining helps\n- do other strategies for negative sample work\n- how important is not to rerank in certain cases (Sec 3.3)\n\nIn conclusion, I encourage the authors to develop the idea further. Taking in into account the issues with the method (or its presentation) and the experimental weaknesses I recommend reject for now.\n\nTypos:\n- [CLS] is not defined in text\n- Shaw et al. should be in Previous methods in Table 3", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "Byx39qufKS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a reranking architecture with a LogicForm-to-NaturalLanguage preprocessing step for semantic parsing. The authors experiment their method on three datasets and get the state of the art results. \n\nThe proposed method is natural. But using neural models to rank (or rerank) is a long-existing technique, regardless of the chosen parametrization of the reranking model. This paper chose BERT. See section-2.6 of this tutorial for more details about using neural models to rank: https://www.microsoft.com/en-us/research/uploads/prod/2017/06/INR-061-Mitra-neuralir-intro.pdf. \n\nOverall, I think the paper is not ready to publish for the following reasons.\n\n1. The method relies much upon manual designs that seem hard to generalize. \n\nBy converting the logic forms to natural languages, the authors can leverage paraphrase datasets and pre-train the critic as a paraphrase model. However, the way they convert the logic forms is different for each dataset and they have to manually design rules for each logic form. \n\n2. It is not clear how certain experimental designs were made. \n\nThe authors chose to not rerank if the candidates' scores are too low or high but close. Such choice and associated thresholds seem arbitrary: how were they actually found out? Were they tuned on a development set? How does the method work if the candidate with the highest score is always picked: in the end, this is what the model is supposed to learn, correct? \n\nOther designs include beam size, whether or not to use a pretrained model, etc. How were such decisions made? Tuned on a development set? \n\n3. The results are not sound enough. Given the issued pointed out in 1 and 2, I am not sure if the results are really sound as the authors claimed. \n\nFor example, what if the authors don’t use a LogicForm-to-NaturalLanguage conversion? What is the result if we directly learn to match input and logic forms? \n\nMoreover, the authors better answer questions in 2 so I can gauge if their hyper-parameters were chosen in the principled ways. Once those are answered, a significant test had better be done since the improvement seems small. \n\n4. Claiming Shaw et al. 2019 in table-3 as ``our methods’’ is wrong. It is clear that Shaw et al. (2019) didn't experiment on OVERNIGHT dataset, but setting up the baseline on a dataset should not be classified as ``our method’’. \n\nMoreover, I have some comments on the model and experiments. These are not weakness, but I think some work in this direction may help improve the paper. \n\n1. The model architecture should be better justified. In its current form, the two arguments (input query and output sequence translated from a logic form) are interchangeable. Why so? Why isn’t an asymmetric architecture more natural? How can the authors use a pair of logic forms as negative examples (in figure-2)? Why do the authors use the Quora dataset in particular? \n\n2. The error analysis might be better to be a bit more quantitative. Its current form doesn’t seem to give insight on how the proposed method really helps. What the authors can do is: you can sample some sentences from the test/development set and count how many comparative words are misused in the original model, among which how many are corrected by reranking.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "BkgMQmijtH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a framework for semantic parsing, which includes a neural generator that synthesizes the logical forms from natural language utterances, and a neural reranker that re-ranks the top predictions generated by beam search decoding using the neural generator. While the neural generator is the same as prior work, the main novelty is the reranker design, which is a binary classifier that takes a pair of natural language utterance/logical form, and predicts the similarity between them. This reranker could also be pre-trained using auxiliary data sources, e.g., Quora question pairs benchmark for paraphrasing. They evaluate their approach on 3 semantic parsing datasets (GEO, ATIS, and OVERNIGHT), and show that their reranker can further improve the performance of the base generator.\n\nI think the general motivation of the framework is sound. Although the idea of reranking is not new in the semantic parsing community, with the most recent work [1] already shows the promise of this direction, the concrete approach described in this paper is different, seems simple yet effective. The most interesting part is to transform the generated logical form into a pseudo-natural language text, so that it becomes a paraphrase of the input natural language utterance in some sense, which enables the re-ranker to be pre-trained with auxiliary data sources, and to use the wordpiece tokenizer that is effective in understanding natural language. In their evaluation, they indeed show that this transformation helps improve the performance of the reranker.\n\nMy main concern of  this paper is about evaluation. First, although they already evaluate on 3 datasets, all of them are not among the most challenging benchmarks in semantic parsing. In [1], they also evaluate on Django and Conala, which are 2 benchmarks to translate natural language to Python, and also are more complicated than the benchmarks in this paper. It would be helpful for the authors to show results on such datasets that the results of baseline neural generators are less satisfactory, which may also make more room for the possible improvement using a re-ranker.\n\nOn the other hand, they also lack a comparison with existing re-ranking approaches. For example, it will be helpful to compare with [1], given that they also evaluate on GEO and ATIS. Right now the results are not directly comparable because: (1) the base generators are different; and (2) the beam size used in this paper (10) is larger than the beam size (5) in [1]. It will be helpful if the authors can at least provide results with a smaller beam size, and would be better if they can provide results that are directly comparable to [1].\n\n[1] Yin and Neubig, Reranking for Neural Semantic Parsing, ACL 2019.\n\n------------\nPost-rebuttal comments\n\nI thank the authors for the response. However, I don't think my concerns are addressed; e.g., without a comparison with previous re-ranking methods, it is hard to justify their proposed approach, given that other re-ranking methods are also able to improve over an existing well-performed generator. Therefore, I keep my original assessment.\n------------", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents and evaluates a technique for semantic parsing, and in particular proposes a model to re-rank the candidates generated by beam search. The paper was reviewed by 3 experts and received Reject, Weak Reject, and Weak Reject opinions. The reviews identified strengths of the paper but also significant concerns, mostly centered around the experimental evaluation (including choice of datasets, lack of direct comparison to baselines, need for more methodical and quantitative analysis, need for additional analysis, etc.) and some questions about the design of the technical approach. The authors submitted responses that addressed some of these concerns, but indicated that additional experimentation would be needed to address all of them. In light of these reviews, we are not able to recommend acceptance at this time, but I hope authors use the detailed, constructive feedback to improve the paper for another venue.
Decision: Reject