Improving Semantic Parsing with Neural Generator-Reranker Architecture

1 INTRODUCTION . Semantic parsing is the task of deriving machine interpretable meaning representations such as logical forms or structured queries from natural language utterances . These meaning representations can be executed in various environments , making semantic parsing applicable in many frameworks such as querying data/knowledge bases for question answering ( Zelle & Mooney , 1996 ; Zettlemoyer & Collins , 2007 ; Liang et al. , 2011 ; Berant et al. , 2013 ) , generating regular expression ( Kushman & Barzilay , 2013 ) , instruction following ( Artzi & Zettlemoyer , 2013 ) , and communicating with robots ( Chen & Mooney , 2011 ; Tellex et al. , 2011 ; Bisk et al. , 2016 ) . Conventionally , semantic parsing has been done with a two step approach : first , a large number of potential candidates are generated using deterministic methods and combinatorial search ( the generator ) , and then the best candidate is selected among them with a probabilistic method or scoring ( the critic ) ( Berant & Liang , 2014 ; Kwiatkowski et al. , 2013 ; Zettlemoyer & Collins , 2005 ; Berant et al. , 2013 ; Cai & Yates , 2013 ) . With recent advancement of neural networks , neural models with encoder-decoder architectures has obtained impressive improvements ( Dong & Lapata , 2016 ; Jia & Liang , 2016 ; Herzig & Berant , 2017 ; Su & Yan , 2017 ; B. Chen & Han , 2018 ; Shaw et al. , 2019 ) . These encoder-decoder based neural semantic parsers produce one candidate given an input sentence , essentially acting as both the generator and the critic . However , undesirable prediction errors still occur , e.g . predicting a wrong comparative or superlative structure ( such as < instead of ≤ ) . On the other hand , neural semantic parsing models have a high recall , i.e . top-n predictions of the model cover the gold-standard meaning representation most of the time ( c.f . Table 2 in Section 2 ) . In this work we propose a generator-reranker architecture that uses two neural networks for semantic parsing : a generator network , which generates a list of potential candidates , and a reranker system , which consists of a pre-processing step for the candidates followed by a novel critic network that reranks these candidates based on the similarity between each candidate and the input sentence . An advantage of separating the semantic parsing process into a generator network and a critic network is that the critic observes each candidate and the input sentence entirely , taking into account bidirectional representations of both sentences and can globally reason over the entire candidate . This may be more effective in terms of choosing the right candidate and mitigating some of the errors arising from auto-regressive decoding in the generator . Furthermore , the critic can leverage extra data sources for training , e.g . a paraphrase dataset and allow for better transfer learning . Our key contributions in this work are the following : 1 . We introduce a neural critic model , which reranks the candidates of a semantic parser based on their similarity score with respect to the input utterance . 2 . We propose various pre-processing methods for the candidates to leverage the pre-trained representations of their tokens using the critic model . 3 . We show through extensive qualitative and quantitative studies how the critic model helps mitigating errors of a state-of-the-art neural semantic parser and improves the performance . Evaluation results of our approach on three existing semantic parsing datasets ( see Table 1 for a sample input-output pair for each dataset ) show that our model improves upon the state-of-the-art results and the generator-reranker architecture can substantially improve parsing performance . 2 RELATED WORK . The semantic parsing problem has received significant attention and has a rich literature ( Kamath & Das , 2018 ) . While traditional approaches ( Kate & Mooney , 2006 ; Wong & Mooney , 2007 ; Clarke et al. , 2010 ; Zettlemoyer & Collins , 2007 ; Kwiatkowski et al. , 2011 ; Wang et al. , 2015 ; Li et al. , 2015 ; Cai & Yates , 2013 ; Berant et al. , 2013 ; Quirk et al. , 2015 ; Artzi et al. , 2015 ; Zhang et al. , 2017 ) rely on high-quality lexicons , manually-built templates , and/or domain or representation specific features , in more recent studies neural models with encoder-decoder architectures show impressing results ( Dong & Lapata , 2016 ; Jia & Liang , 2016 ; Herzig & Berant , 2017 ; Su & Yan , 2017 ; B. Chen & Han , 2018 ; Shaw et al. , 2019 ) . Among recent results , Herzig & Berant ( 2017 ) achieves the best performance on the OVERNIGHT dataset ( Wang et al. , 2015 ) , which consists of 8 various domains . The model in Herzig & Berant ( 2017 ) is an attention-based sequence-to-sequence model and it is trained jointly over 8 domains . Very recently , Shaw et al . ( 2019 ) presents an approach that uses a Graph Neural Network ( GNN ) architecture , which successfully incorporates information about relevant entities and their relations in parsing natural utterances . Similar to Vinyals et al . ( 2015 ) ; Jia & Liang ( 2016 ) ; Herzig & Berant ( 2017 ) , the decoder has a copying mechanism , which can copy an entity to the output during parsing . This model achieves the best performance in GEO dataset ( Zelle & Mooney , 1996 ) and is competitive with state-of-the-art in ATIS dataset ( Dahl et al. , 1994 ) . We note that this model was not applied to the OVERNIGHT dataset . We provide in Table 2 the comparison between top-10 ( 25 ) oracle1 and top-1 ( greedy decoding ) accuracy for the state-of-the-art sequence-to-sequence models . We note that while it may not be possible to reach top-10 ( 25 ) accuracy for all datasets2 , there is certainly room for improvement . 1The fraction of test examples where the gold-standard logical form is present in one of the top-10 ( 25 ) predicted logical forms . 2For instance , Wang et al . ( 2015 ) reports 17 % rate of incorrect labels in the OVERNIGHT dataset . Motivated by these observations , we consider a generator-reranker architecture for semantic parsing . The reranker system consists of a critic network , which reranks candidate logical forms based on their similarity scores to the input utterance . We propose various techniques for processing candidate logical forms before the critic scores their similarity to the input utterance . For the generator , we use the model in Shaw et al . ( 2019 ) and for the critic , we use the BERT model ( Devlin et al. , 2018 ) for all three datasets . Although the setting is parser agnostic , we would like to investigate if the generator-reranker architecture can further improve the performance of the best performing parser . To the best of our knowledge , our reranker system is novel and has not been considered before in semantic parsing . However , the idea of reranking the candidates of a generator model appears in various applications in the literature . We refer the reader to Collins & Koo ( 2005 ) for reranking based on a combination of feature functions for various NLP tasks . For semantic parsing task , Yavuz et al . ( 2016 ) applies reranking based on predicted answer type . We note that this method may not be effective for the cases where the candidate logical forms return the same answer type but the top prediction is wrong ( such as predicting a wrong comparative , e.g . < instead of ≤ ) . We believe choosing the best candidate by assessing its relevance to the input utterance entirely would be a more effective method . In a more recent work by Yin & Neubig ( 2019 ) , reranking is applied by two main quality-measuring features of candidate logical forms . The first one is the reconstruction feature , using the probability of reproducing the original input utterance x from y . The second is the discriminative matching feature , which is based on pair-wise associations of tokens in x and y . In our work , reranking of candidate logical forms is applied based on their similarity with the input utterance directly using a critic model which can leverage the pre-trained representations of processed logical forms as well as extra data sources to learn similarity . 3 MODEL ARCHITECTURE . Our goal is to learn a model which maps an input utterance x to a logical form representation of its meaning y . The input utterance x is a sequence of words x1 , x2 , . . . , xn1 ∈ V ( in ) where V ( in ) is the input vocabulary and the output logical form y is a sequence of tokens y1 , y2 , . . . , yn2 ∈ V ( out ) where V ( out ) is the output vocabulary . In this work , we consider a combination of two models for semantic parsing . Figure 1 illustrates the model architecture along with an example . We describe the model architecture in what follows . 3.1 GENERATOR . To generate candidate logical forms , we use the model recently introduced in Shaw et al . ( 2019 ) , which is based on the Transformer architecture ( Vaswani et al. , 2017 ) , with the self-attention layer extended to incorporate relations between input elements , and the decoder extended with a copy mechanism similar to Vinyals et al . ( 2015 ) ; Jia & Liang ( 2016 ) ; Herzig & Berant ( 2017 ) . The model is trained with natural utterances paired with logical forms and learns shared representations from these pairs . Candidate logical forms are generated using beam search for a given input utterance . 3.2 PROCESSING CANDIDATES . Before scoring the similarity between a logical form and an input utterance , we preprocess the candidate logical forms . We propose the processing methods in an increasing order of their complexity and how close they map the logical forms to natural text . Note that the critic can leverage the pretrained representations of more processed logical forms better and can be more effective when scoring similarities . Figure 3 in Appendix A illustrates these methods along with an example . 3.2.1 RAW LOGICAL FORM . In this method , we consider raw logical forms without any processing for calculating similarity with respect to the input utterance . This is the simplest method that is applicable to any dataset . 3.2.2 NATURAL LANGUAGE ENTITY NAMES . In this method , we simply convert entities to natural text . The output tokens are often selfexplanatory and easy to be converted to simple text , e.g. , “ num assists ” → “ number of assists ” , “ en.location.greenberg cafe ” → “ greenberg cafe ” in OVERNIGHT dataset , and “ arrival time ” → “ arrival time ” in ATIS dataset etc . This approach requires an additional step but can be applied in a straightforward manner . The advantage of this approach is that the critic can leverage pre-trained representations for these tokens . We describe the exact procedure in Appendix A for each dataset . 3.2.3 TEMPLATED EXPANSIONS . In this method , logical forms are converted to canonical utterances using a deterministic template ( e.g . arg max ( type.player , numRebounds ) to “ player that has the largest number of rebounds ” ) . The purpose of the canonical utterances is to capture the meaning of the logical forms . Here the assumption is that while it is nearly impossible to generate a grammar that parses all utterances , it is possible to write one that generates one canonical utterance for each logical form ( Wang et al. , 2015 ) . We use this method only for OVERNIGHT dataset as there is an available template introduced in Wang et al . ( 2015 ) to convert the logical forms to canonical utterances . In this case , the critic can be considered as a paraphrase model , producing a score based on the equivalence of two sentences semantically . During test time , candidate logical forms that do not have a corresponding canonical utterance are treated as incorrect candidates and they are not included in ranking .

Summary: This paper proposes a framework for semantic parsing, which includes a neural generator that synthesizes the logical forms from natural language utterances, and a neural reranker that re-ranks the top predictions generated by beam search decoding using the neural generator. While the neural generator is the same as prior work, the main novelty is the reranker design, which is a binary classifier that takes a pair of natural language utterance/logical form, and predicts the similarity between them. This reranker could also be pre-trained using auxiliary data sources, e.g., Quora question pairs benchmark for paraphrasing. They evaluate their approach on 3 semantic parsing datasets (GEO, ATIS, and OVERNIGHT), and show that their reranker can further improve the performance of the base generator.