Syntactic Question Abstraction and Retrieval for Data-Scarce Semantic Parsing

tating complex logical forms is costly . Here , we propose SYNTACTIC QUESTION ABSTRACTION & RETRIEVAL ( SQAR ) , a method to build a neural semantic parser that translates a natural language ( NL ) query to a SQL logical form ( LF ) with less than 1,000 annotated examples . SQAR first retrieves a logical pattern from the train data by computing the similarity between NL queries and then grounds a lexical information on the retrieved pattern in order to generate the final LF . We validate SQAR by training models using various small subsets of WikiSQL train data achieving up to 4.9 % higher LF accuracy compared to the previous state-of-the-art models on WikiSQL test set . We also show that by using query-similarity to retrieve logical pattern , SQAR can leverage a paraphrasing dataset achieving up to 5.9 % higher LF accuracy compared to the case where SQAR is trained by using only WikiSQL data . In contrast to a simple pattern classification approach , SQAR can generate unseen logical patterns upon the addition of new examples without re-training the model . We also discuss an ideal way to create cost efficient and robust train datasets when the data distribution can be approximated under a data-hungry setting . 1 . Introduction . Semantic parsing is the task of translating natural language into machine-understandable formal logical forms . With the help of recent advance in deep learning technology , neural semantic parsers have achieved state-of-the-art results in many tasks [ Dong and Lapata , 2016 , Jia and Liang , 2016 , Iyer et al. , 2017b ] . However , their training requires the preparation of a large amount of labeled data ( questions and corresponding logical forms ) which is often not scalable due to the requirement of expert knowledge necessary in writing logical forms . Here , we develop a novel approach SYNTACTIC QUESTION ABSTRACTION & RETRIEVAL ( SQAR ) for semantic parsing task under data-hungry setting . The model constrains the logical form search space by retrieving logical patterns from the train set using natural language similarity with assistance of a pre-trained language model . The subsequent grounding module only needs to map the retrieved pattern to the final logical form . We evaluate SQAR on various subsets of WikiSQL train data [ Zhong et al. , 2017 ] consisting of 850∼2750 samples which occupies 1.5–4.9 % of the full train data . SQAR shows up to 4.9 % higher logical form accuracy compared to the previous best open sourced model SQLOVA [ Hwang et al. , 2019 ] . Also , we show that natural language sentence similarity dataset can be leveraged in SQAR by pre-training the backbone of SQAR using Quora pharaphrasing data which results in up to 5.9 % higher logical form accuracy . In general , the retrieval approach causes the limitation on dealing with unseen logical patterns . In contrast , we show that SQAR can generate unseen logical patterns by collecting new examples without re-training opening an interesting possibility of generalizable retrieval-based semantic parser . Our contributions are summarized as follows : • Compared to the previous best open-sourced model [ Hwang et al. , 2019 ] , SQAR achieves the state-of-the-art performance on the WikiSQL test data under data-scarce environment . • We show that SQAR can leverage natural language query similarity datasets to improve logical form generation accuracy . • We show that retrieval-based parser can handle unseen new logical patterns on the fly without re-training . • For the maximum cost-effectiveness , we find that it is important to carefully design the train data distribution , not merely following the ( approximated ) data distribution . 2 . Related work . WikiSQL [ Zhong et al. , 2017 ] is a large semantic parsing dataset consisting of 80,654 natural language utterances and corresponding SQL annotations . Its massive size has invoked the development of many neural semantic parsing models [ Xu et al. , 2017 , Yu et al. , 2018 , Dong and Lapata , 2018 , Wang et al. , 2017 , 2018 , McCann et al. , 2018 , Shi et al. , 2018 , Yin and Neubig , 2018 , Xiong and Sun , 2018 , Hwang et al. , 2019 , He et al. , 2019 ] . Berant and Liang [ Berant and Liang , 2014 ] built the semantic parser that uses the query similarity between an input question and paraphrased canonical natural language representations generated from candidate logical forms . In our study , candidate logical forms and corresponding canonical forms do not need to be generated as input questions are directly compared to the questions in the training data , circumventing the burden of full logical form generation . Dong and Lapata [ Dong and Lapata , 2018 ] developed the two step approach for logical form generation , similar to SQAR using sketch representation as intermediate logical forms . In SQAR , intermediate logical forms are retrieved from train set using question similarity being specialized for data-hungry setting . Finegan-Dollak et al . [ Finegan-Dollak et al. , 2018 ] developed the model that first finds corresponding logical pattern and fills the slots in the template . While their work resembles SQAR , there is a fundamental difference between two approaches . The model from [ Finegan-Dollak et al. , 2018 ] classifies input query into logical pattern whereas we use query-to-query similarity to retrieve logical pattern non-parametrically . By retrieving logical pattern using the similarity in natural language space , paraphrasing datasets can be employed during training which is relatively easy to label compared to semantic parsing datasets . Also , in contrast to classification methods , SQAR can handle unseen logical patterns by including new examples into the train set without re-training the model during inference stage ( see section . 5.5 ) . Also our focus is developing competent model with small amount of data which has not been studied in [ Finegan-Dollak et al. , 2018 ] . Hwang et al . [ Hwang et al. , 2019 ] developed SQLOVA that achieves state-of-the-arts result in the WikiSQL task . SQLOVA consits of table-aware BERT encoder and NL2SQL module that generate SQL queries via slot-filling approach . 3 . Model . The model generates the logical form L ( SQL query ) for a given NL query Q and its corresponding table headers H ( Fig . 1 ) . First , the logical pattern l is retrieved from the train set by finding the most similar NL query with Q . For example in Fig . 1 , Q is “ What is the points of South Korea player ? ” . To generate logical form L , SQAR retrieves logical pattern l = SELECT # 1 WHERE # 2 = # 3 by finding the most similar NL query from the train set , for instance [ “ Which fruit has yellow color ? ” , SELECT Fruit WHERE Color = Yellow ] . Then # 1 , # 2 , and # 3 in l are grounded to Point , Country , and South Korea respectively by the grounding module using information from Q and table headers . The process is depicted schematically in Fig . 2a . The detail of each step is explained below . 3.1 Syntactic Question Abstractor . The syntactic question abstractor generates two vector representation q and g of an input NL query Q ( Fig . 2b ) . q is trained to represent syntactic information of Q and used in the retriever module ( Fig . 2c ) . g is trained to represent lexical information of Q by being used in the grounder ( Fig . 2d ) . The logical patterns of the WikiSQL dataset consist of combination of six aggregation operators ( none , max , min , count , sum , and avg ) , and three where operators ( = , > , and < ) . The number of conditions in where clause is ranging from 0 to 4 . Each condition is combined by and unit . In total , there are 210 possible SQL patterns ( 6 select clause patterns × 35 where clause patterns , see Fig . A1 ) . To extract these syntactic information from NL query , both an input NL query Q and the queries in train set { Qt , i } are mapped to a vector space ( represented by q and { qt , i } , respectively ) via table-aware BERT encoder [ Devlin et al. , 2018 , Hwang et al. , 2019 ] ( Fig . 2b ) . The input of the encoder consists of following tokens : [ CLS ] , E , [ SEP ] , Q , [ SEP ] , H , [ SEP ] where E stands for SQL language element tokens such as [ SELECT ] , [ MAX ] , [ COL ] , · · · ) separated by [ SEP ] ( a special token in BERT ) , Q represents question tokens , and H denotes the tokens of table headers in which each header is separated by [ SEP ] . E is included to contextualize and use them during grounding process ( section 3.3 ) . Segment ids are used to distinguish E ( id = 0 ) from Q ( id = 1 ) and H ( id = 1 ) as in BERT [ Devlin et al. , 2018 ] . Next , two vectors q ≡ v [ CLS ] 0 : dq and g ≡ v [ CLS ] dq : ( dq+2dh ) are extracted from the ( linearly projected ) encoding vector of [ CLS ] token where i : j notation indicates the elements of vector between ith and jth indices . In this study , dq = 256 and dh = 100 . 3.2 Retriever . To retrieve logical pattern of Q , the questions from the train set ( { Qt , i } ) are also mapped to the vector space ( { qt , i } ) using the syntactic question abstractor . Next , the logical pattern is found by measuring Euclidean L2 distance between q and { qt , i } . qt , i∗ = argmin qt , i ||q − qt , i||L2 ( 1 ) Since qt , i∗ has corresponding Qt , i∗ and logical form Lt , i∗ , the logical pattern l∗ can be obtained from Lt , i∗ after delexicalization . The process is depicted in Fig . 2c . In SQAR , maximum 10 closest qt , i∗ are retrieved and the most frequently appearing logical pattern is selected for the subsequent grounding process . SQAR is trained using the negative sampling method . First , one positive sample ( having the same logical pattern with input query Q ) , and 5 negative samples ( having different logical pattern ) are randomly sampled from the train set . Then six L2 distances are calculated as above and interpreted as approximate probability by using softmax function after multiplied by -1 . The cross entropy function is employed for the training . 3.3 Grounder . To ground retrieved logical pattern l∗ , following LSTM-based pointer network is used [ Vinyals et al. , 2015 ] . Dt = LSTM ( Pt−1 , ( ht−1 , ct−1 ) ) h0 = g0 : dh c0 = gdh:2dh st ( i ) =W ( WHi +WDt ) pt ( i ) = softmax st ( i ) , ( 2 ) where Pt−1 stands for the one-hot vector ( pointer to the input token ) at time t−1 , ht−1 and ct−1 are hidden- and cell-vectors of the LSTM decoder , W ’ s denote ( mutually different ) affine transformations , and pt ( i ) is the probability of observing ith input token at time t. Here dh ( =100 ) is the hidden dimension of the LSTM . Compared to a conventional pointer network , our grounder has three custom properties : ( 1 ) as logical pattern is already found from the retriever , the grounder does not feed the output as the next input when the input token is already present in the logical pattern whereas lexical outputs like column and where values are fed into the next step as an input ( Fig . 2d ) ; ( 2 ) to generate conditional values for where clause , the grounder infers only the beginning and the end token positions from the given question to extract the condition values for where clause ; ( 3 ) the multiple generation of same column on where clause is avoided by constraining the search space . The syntactic question abstractor , the retriever , and the grounder are together named as SYNTACTIC QUESTION ABSTRACTION & RETRIEVAL ( SQAR ) .

Summary: The paper presents a retrieval-based semantic parsing method for the data-scarce setting. The model retrieves a logical pattern from the train data by computing the similarity between NL queries. Then lexicons are added to the retrieved pattern in order to generate the final LF. The motivation and the proposed method make sense to me. The experimental results also show that the approach improves the performance over several baseline methods. The only concern is that the experiments are only conducted on wikisql, which is not very data-hungry. The paper reduces the number of training set to simulate the setup. The paper could be improved by conducting experiments on more small-scale datasets.