SCoRe: Pre-Training for Context Representation in Conversational Semantic Parsing

1 INTRODUCTION . The goal of task-oriented dialog systems is to assist the user in completing a certain task by performing an action or retrieving relevant information ( Tur & Mori , 2011 ) . They are often built on top of a structured ontology grounded in a knowledge base , a database , or a set of API calls . This in contrast to open-domain dialog systems ( also referred to as chit-chat systems ) where the goal is to maximize engagement with users in open-ended conversations ( Jafarpour et al. , 2010 ; Ritter et al. , 2011 ) . A key component of task-oriented conversational systems is Conversational Semantic Parsing ( CSP ) , which converts each utterance in the dialog into a formal language query ( e.g. , SQL , SPARQL ) that can be executed against the structured ontology . CSP has been extensively studied in several academic and industrial research settings such as dialog systems ( e.g. , dialog state tracking in MWOZ ( Budzianowski et al. , 2018 ) ) , interacting with physical agents ( e.g. , ( Chai et al. , 2018 ) ) , context-dependent semantic parsing ( e.g. , SPARC ( Yu et al. , 2019b ) ) , SQL-grounded state tracking ( e.g. , COSQL ( Yu et al. , 2019a ) ) , and sequential question answering ( e.g. , SQA ( Iyyer et al. , 2017 ) ) . These settings differ in some respect , but they share the same overall objective and key challenge : how to jointly represent the natural language utterances and underlying structured ontology while taking into consideration the multi-turn dynamics of the dialog . Similar to many other natural language tasks , recent work in CSP has significantly benefited from advances in language model pre-training . However , existing general-purpose pre-trained language models , e.g . BERT ( Devlin et al. , 2019 ) , are pre-trained on free-form text data using language model objectives . This limits their ability in modeling the structural context or the multi-turn dynamics of the dialogs . This presents an opportunity to improve pre-trained LMs to specifically address these limitations for CSP tasks . Recent work has demonstrated the benefits of adapting pre-trained LMs Published as a conference paper at ICLR 2021 Transformer Encoder < s > also show the names of their publishers < s > who are … authors < s > find < mask > … books < /s > author id < /s > author name < /s > … ... < /s > sale amount Current Question Dialog History Database Schema < s > TCS < s > TCS < /s > CCS < /s > CCS < /s > CCS < /s > CCS < mask > MLM to specific domains ( Gururangan et al. , 2020 ) or tasks ( Zhang et al. , 2019b ) via a second phase of pre-training . For example , open-domain dialogue language models such as DialoGPT ( Zhang et al. , 2020 ) and ConveRT ( Henderson et al. , 2019 ) are pre-trained on the Reddit data and applied to dialog response generation and retrieval tasks . In this paper , we introduce SCORE ( Structured & Sequential Context Representation ) , a language model pre-training approach for CSP tasks . SCORE adapts general pre-trained LMs by introducing a second phase of pre-training using multiple objectives that capture both multi-turn dynamics and the structural contexts in a dialog . In contrast to open-domain dialogs , CSP datasets are usually much smaller due to the difficulty and expense of obtaining and labeling data ( mapping natural language utterances to formal language ) . Unlike most prior work on contextualized LMs which are pre-trained on free text , according to the finding where questions in CSP tasks are more compositional than other free-text since they can be mapped into formal representations , we propose to train SCORE on synthesized conversational semantic parsing data with multiple training objectives that aim to ground utterances into the schema of the underlying ontology and to model the relationship between different utterances in the multi-turn conversation . In this way , SCORE can effectively inject structural and conversational inductive biases in LMs that can translate to many CSP tasks . SCORE uses an order of magnitude smaller dataset for the second stage of pre-training , does not require changes to the pre-trained model architecture , can be used as a drop-in replacement of general pre-trained LMs with any semantic parsing model , and can be used out-of-the-box in many CSP tasks . We apply SCORE to four different CSP tasks : ( 1 ) sequential text-to-SQL ( SPARC ) , ( 2 ) conversational text-to-SQL ( COSQL ) , ( 3 ) dialog state tracking ( MWOZ ) , and ( 4 ) weakly-supervised sequential question answering ( SQA ) . The fours tasks represent different scenarios , types of ontologies , supervision signals , system responses , and domains ( see Table 1 for a detailed comparison and Figure 1 for examples ) . We demonstrate that : ( 1 ) SCORE training objectives can effectively incorporate synthesized data , ( 2 ) a single pre-trained SCORE model can be used for several CSP tasks and can be combined with many baseline systems with different model architectures and ( 3 ) SCORE significantly improve all baseline systems and achieves new state-of-the-art results on three benchmarks ( SPARC , SPARC , and MWOZ ) and comparable performance to state-of-the-art results on the fourth ( SQA ) . 2 APPROACH . The key challenge of CSP is to capture the relationship between the natural language utterance and the structured ontology in the multi-turn dialog dynamics . To this end , we inject structural and conversational inductive biases in SCORE by introducing two objective functions : Column Contextual Semantics ( CCS ) and the Turn Contextual Switch ( TCS ) . Because the size of existing semantic parsing datasets is limited , we produce synthesized data for pretraining SCORE by sampling from the context-free grammar induced from complex text-to-SQL examples in different domains . Moreover , to prevent SCORE from overfitting to the linguistic pattern of our synthesized data , we use the Masked Language Modeling ( MLM ) objective on human-generated utterances as regularization . 2.1 PRELIMINARIES . Task Definition In CSP , at each turn t , we aim to produce a formal representation qt given the current utterance ut , the interaction history ht = [ u1 , u2 , . . . , ut−1 ] , and the schema c ( table and column names , slots , etc . ) of the target database ( ontology ) d. To cover different problem variants , we consider four popular CSP tasks shown in Table 1 : SPARC ( sequential text-to-SQL ) , COSQL ( conversational text-to-SQL ) , MWOZ ( dialogue state tracking ) , and SQA ( weakly supervised sequential question answering ) . They have different target formal language and structured ontology : • For the utterance u , it is the user question for SPARC and SQA , while for COSQL and MWOZ , u is the combination of a user query and a system response . • For the database d , SPARC and COSQL use multi-table databases ; for MWOZ , the pre-defined ontology d can also be viewed as a database ; for SQA , d is a single table . • For the formal representation q , it is the SQL query for SPARC and COSQL ; in MWOZ it is the slot-value pairs that can be viewed as simple SQL queries consisting of SELECT and WHERE clauses ; and for SQA , q is the latent program . Base Architecture The base architecture of SCORE takes as input a single turn of a CSP dialog 〈ut , ht〉 jointly with the underlying database schema c. Given this contextualized conversational input Ct = 〈ut , ht , c〉 , SCORE encodes it into contextualized conversation representations ~St for each token in Ct . The encoder architecture follows RoBERTa ( Liu et al. , 2019b ) . It is then followed by a linear layer and normalized ( Ba et al. , 2016 ) to produce final representations ~ht for each token : Ct = 〈ut , ht , c〉 , ~St = ROBERTA ( Ct ) , ht , i = LayerNorm ( GELU ( W1St , i ) ) ∀St , i ∈ ~St , ( 1 ) where GELU is an activation by Hendrycks & Gimpel ( 2016 ) and W1 is a learned parameter matrix . To build Ct , we first concatenate current utterances ut and dialog history ht separated by a special token < s > , as this simple strategy has been shown effective in state-of-the-art CSP systems ( Zhang et al. , 2019c ; Wu et al. , 2019 ; Liu et al. , 2020 ; Heck et al. , 2020 ) . To incorporate the database schema , we follow Hwang et al . ( 2019 ) to concatenate all column names as a single sequence . Column names are separated by the special token < /s > and prefixed by their corresponding table name . 2.2 SCORE PRE-TRAINING . SCORE addresses the challenges of CSP by pre-training a task-oriented language model contextualized by the conversational flow and the underlying ontology . In pre-training , the SCORE model is self-supervised by two novel objectives in addition to the established Masked Language Modeling ( MLM ) objective . These objectives facilitate the accurate representation of the conversational flow between dialog turns and how this flow maps to the desired columns in the ontology . Column Contextual Semantics The first challenge of CSP is capturing the alignment between the natural language utterance and the underlying database schema . To address it , we optimize the SCORE model with the auxiliary objective of Column Contextual Semantics ( CCS ) . For each column in the database schema c , CCS targets the operations that should be performed on this column in a given conversational turn . Specifically , each formal representation q is decomposed into operations on columns and tables , e.g . GROUP BY and HAVING for SQL queries , or WHERE for the slot-value pairs . In this way , our data covers 148 column operations . We use the encoding of the special token < /s > right before each column or table name to predict its corresponding operations , and then compute the CCS loss : LCCS ( Ct ) = ∑ i∈c CrossEntropy148 ( LayerNorm ( W2 h c t , i ) , CCS ( qt ) ) ( 2 ) where hct , i is the contextualized representation of the i th column ’ s special token < /s > in the contextualized input Ct , CCS ( qt ) returns the column operation label for the current formal representation qt , CrossEntropy148 computes the 148-way cross-entropy between the column operation prediction and label , and W2 is a learned parameter matrix . Usr : Find the names of the top 3 highest sales books . Usr : Who are their authors ? Usr : Also show the names of their publishers . … ... Usr : I am looking for a cheap restaurant in the centre of the city Sys : There is a cheap chinese restaurant called Dojo Noodle Bar . Usr : Yes please , for 8 people at 18:30 on Thursday … ... Usr : I also need to book a taxi between to the restaurant at 20:30 . Sys : The taxi is booked . SELECT title FROM book ORDER BY sale_amount DESC LIMIT 3 SELECT t1.title , t1.name FROM author AS t1 JOIN book AS t2 ON t1.id = t2.author_id ORDER BY t2.sale_amount DESC LIMIT 3 SELECT t1.title , t1.name , t3.name FROM author AS t1 JOIN book AS t2 ON t1.id = t2.author_id JOIN press AS t3 ON t2.press_id = t3.id ORDER BY t2.sale_amount DESC LIMIT 3 Restaurant ( Price=cheap , area=center ) Restaurant ( Price=cheap , area=center , name=Dojo Noodle Bar , people=8 , time=18:30 , day=Thursday ) Restaurant ( price=cheap , area=center , name=Dojo Noodle Bar , people=8 , time=18:30 , day=Thursday ) Taxi ( leaveAt=20:30 , destination=Dojo Noodle Bar ) Restaurant : name | price | area | … | time … ... Taxi : leaveAt | … | destination D ia lo g st at e tra ck in g Author : id | name | ... | country Press : id | name | ... | address … ... Book : id | title | author id | … | sale_amount Transformer Encoder Dialogs DatabaseFormal Programs M ul titu rn te xt -to -S Q L < s > also show the names of their publishers < s > who are … authors < s > find < mask > … books < /s > author id < /s > author name < /s > … ... < /s > sale amount Current Question Dialog History Database Schema < s > TCS INS ( SELECT.column ) < s > < /s > < /s > < /s > < /s > < mask > TCS INS ( SELECT.column ) MLM the CCS ... CCS SELECT CCS ORDER BY DESC LIMIT CCS None Figure 2 : Pre-training of a SCORE encoder on a SPARC text-to-SQL example from Figure 1 . Turn Contextual Switch The second challenge of CSP is capturing the conversational context flow and how it is grounded into the formal representations . The TCS objective aims to capture this grounding of context flow . To this end , it rgets predicting the difference in formal representations between dialog turns based on the natural language utterance . Based on the context-free grammar of SQL , we identify 26 possible turn difference oper tions that a conversatio l turn could elicit . They encod changes between different turns of user queries ( the system response is not involved here ) since we assume that most turn contextual shifts are from the user . For example , INS ( WHERE ) indicates inserting a new WHERE condition and DEL ( SELECT.agg ) indicates removing an aggregate operation from a SELECT statement ( e.g . when an utterance “ Show all the ages instead. ” elicits a change SELECT MAX ( age ) ... → SELECT age ... ) . We use the encoding of the special token < /s > right before each turn to predict the context switch label between this turn and the previous history : LTCS ( Ct ) = CrossEntropy26 ( LayerNorm ( W3Hst ) , TCS ( qt , qt−1 ) ) ( 3 ) where Hst ∈ R ( t−1 ) ×d is the contextualized representation of all previous turns in Ct with hidden dimension d , TCS ( qt , qt−1 ) returns the turn difference operations from qt−1 to qt , and W3 is a learned parameter matrix . We don ’ t use this objective to pre-train SCORE for MWOZ because the context switch label between turns is relatively simple in MWOZ ( only select and where changes ) . Masked Language Modeling As in prior work on large-scale language models ( Devlin et al. , 2019 ) , we use the Masked Language Modeling ( MLM ) objective to facilitate contextual representation learning for natural language utterances . Importantly for regularization , we only apply this loss on in-domain human-annotated natural language data . Namely , it includes utterances in SPARC , COSQL , and SQA as well as nine task-oriented dialog datasets processed by Wu et al . ( 2020 ) for MWOZ ( see data statistics in Figure 4 ) . Formally , the MLM loss is given by : LMLM ( Ct ) = ∑ m CrossEntropyVocab ( LayerNorm ( W4h m t ) ) ( 4 ) where hmt are the contextualized representations of the masked 15 % of tokens in Ct , and W4 is a learned parameter matrix . Pre-Training Setup and Steps To summarize the pre-training steps , we first collect a dataset Dnat of combined human-annotated natural language questions ( without labels ) from existing CSP tasks ( as mentioned above ) , and create a large synthesized conversational data Dsyn that is generated by a grammar induced from a small set of SPARC annotated examples ( See 2.3 ) . After that , we incorporate both two datasets in pre-training . More specifically , synthetic and natural examples are randomly sampled during pre-training . The total pre-training loss is the sum of the three objectives with CCS and TCS only applied to Dsyn and MLM only to Dnat : L = ∑ Ct∈Dsyn ( LCCS ( Ct ) + LTCS ( Ct ) ) + ∑ Ct∈Dnat LMLM ( Ct ) ( 5 ) Figure 2 shows an overview of SCORE pre-training on an example SPARC dialogue from Figure 1 . We report additional implementation details for pre-training SCORE in Section 3.3 and Appendix C .

Summary: In this paper, the authors proposed a pre-training strategy for Conversational Semantic Parsing (CSP) tasks. The pre-training is run on top of any existing LM (i.e., in this work RoBERTA has been used), and uses three additional loss functions to inject the CSP inductive bias into the LM: Column Contextual Semantics (CCS), Turn Contextual Switch (TCS) and Masked Language Modeling (MLM). Moreover, the authors proposed to use synthetically generated data in the pretraining. The results are presented in four well-know datasets for CSP: SPARC, COSQL, MWOZ, and SQA.