Multi-Task Learning for Semantic Parsing with Cross-Domain Sketch

1 INTRODUCTION . Recently many natural language processing ( NLP ) tasks based on the neural network have shown promising results and gained much attention because these studies are purely data-driven without linguistic prior knowledge . Semantic parsing task which maps a natural language sentence into a machine-readable representation ( Fan et al . ( 2017 ) ) , as a particular translation task , can be treated as a sequence-to-sequence problem ( Dong & Lapata ( 2016 ) ) . Lately , a compositional graph-based semantic meaning representation language ( MRL ) has been introduced ( Kollar et al . ( 2018a ) ) , which converts utterance into logic form ( action-object-attribute ) , increasing the ability to represent complex requests . This work is based on MRL format for semantic parsing task . Semantic parsing highly depends on the amount of annotated data and it is hard to annotate the data in logic forms such as Alexa MRL . Several researchers have focused on the area of multi-task learning and transfer learning ( Hakkani-Tür et al . ( 2016 ) , Fan et al . ( 2017 ) , Kollar et al . ( 2018b ) ) with the observation that while these tasks differ in their domains and forms , the structure of language composition repeats across domains ( Herzig & Berant ( 2017 ) ) . Compared to the model trained on a single domain only , a multi-task model that shares information across domains can improve both performance and generalization . However , there is still a lack of interpretations why the multi-task learning setting works ( Ruder ( 2017 ) ) and what the tasks have shared . Some NLP studies around language modeling ( Le & Mikolov ( 2014 ) , Vaswani et al . ( 2017 ) , Devlin et al . ( 2018 ) ) indicate that implicit commonalities of the sentences including syntax and morphology exist and can share among domains , but these commonalities have not been fully discussed and quantified . To address this problem , in this work , compared to multi-task learning mentioned above which directly use neural networks to learn shared features in an implicit way , we try to define these cross-domain commonalities explicitly as cross-domain sketch ( CDS ) . E.g. , Search weather in 10 days in domain Weather and Find schedule for films at night in domain ScreeningEvent both have action SearchAction and Attribute time , so that they share a same MRL structure like SearchAction ( Type ( time @ ? ) ) , where Type indicates domain and ? indicates attribute value which is copying from the original utterance . We extract this domain general MRL structure as CDS . Inspired by the research of coarse-to-fine ( Dong & Lapata ( 2018 ) ) , we construct a two-level encoder-decoder by using CDS as a middle coarse layer . We firstly use General Network to get the CDS for every utterance in all domains . Then for a single specific domain , based on both utterance and extracted CDS , we decode the final target with advanced attention while CDS can be seen as adding some constraints to this process . The first utterance-CDS process can be regarded as a multi-task learning setup since it is suitable for all utterances across the domains . This work mainly introducing CDS using multi-task learning has some contributions listed below : 1 ) We make an assumption that there exist cross-domain commonalities including syntactic and phrasal similarity for utterances and extract these commonalities as cross-domain sketch ( CDS ) which for our knowledge is the first time . We then define CDS on two different levels ( action-level and attribute-level ) trying to seek the most appropriate definition of CDS . 2 ) We propose a general-to-detailed neural network by incorporating CDS as a middle coarse layer . CDS is not only a high-level extraction of commonalities across all the domains , but also a prior information for fine process helping the final decoding . 3 ) Since CDS is cross-domain , our first-level network General Network which encodes the utterance and decodes CDS can be seen as a multi-task learning setup , capturing the commonalities among utterances expressions from different domains which is exactly the goal of multi-task learning . 2 RELATED WORK . 2.1 SPOKEN LANGUAGE UNDERSTANDING . Traditional spoken language understanding ( SLU ) factors language understanding into domain classification , intent prediction , and slot filling , which proves to be effective in some domains ( Gupta et al . ( 2006 ) ) . Representations of SLU use pre-defined fixed and flat structures , which limit its expression skills like that it is hard to capture the similarity among utterances when the utterances are from different domains ( Kollar et al . ( 2018b ) ) . Due to SLU ’ s limited representation skills , meaning representation language ( MRL ) has been introduced which is a compositional graph-based semantic representation , increasing the ability to represent more complex requests ( Kollar et al . ( 2018a ) ) . There are several different logic forms including lambda-calculus expression ( Kwiatkowski et al . ( 2011 ) ) , SQL ( Zhong et al . ( 2018 ) ) , Alexa MRL ( Kollar et al . ( 2018a ) ) . Compared to fixed and flat SLU representations , MRL ( Kollar et al . ( 2018a ) ) based on a large-scale ontology , is much stronger in expression in several aspects like cross-domain and complex utterances . 2.2 SEQUENCE-TO-SEQUENCE FOR SEMANTIC PARSING . Mapping a natural language utterance into machine interpreted logic form ( such as MRL ) can be regarded as a special translation task , which is treated as a sequence-to-sequence problem ( Sutskever et al . ( 2014 ) ) . Then Bahdanau et al . ( 2015 ) and Luong et al . ( 2015 ) advance the sequence-tosequence network with attention mechanism learning the alignments between target and input words , making great progress in the performance . Malaviya et al . ( 2018 ) explore the attention mechanism with some improvements by replacing attention function with attention sparsity . Besides , to deal with the rare words , Gu et al . ( 2016 ) incorporate the copy mechanism into the encoder-decoder model by directly copying words from inputs . Lately , many researchers have been around improving sequence-to-sequence model itself , in interpreting the sentence syntax information . Eriguchi et al . ( 2016 ) encode the input sentence recursively in a bottom-up fashion . Wu et al . ( 2017 ) generate the target sequence and syntax tree through actions simultaneously . Another aspect which has caught much attention is constrained decoding . Krishnamurthy et al . ( 2017 ) and Post & Vilar ( 2018 ) add some constraints into decoding process , making it more controllable . Dong & Lapata ( 2016 ) use the recurrent network as encoder which proves effective in sequence representation , and respectively use the recurrent network as decoder and tree-decoder . Krishnamurthy et al . ( 2017 ) employ the grammar to constrain the decoding process . Dong & Lapata ( 2018 ) , believe utterance understanding is from high-level to low-level and by employing sketch , improve the performance . 2.3 MULTI-TASK LEARNING . For semantic parsing task especially in MRL format , it is expensive and time-consuming to annotate the data , and it is challenging to train semantic parsing neural models . Multi-task learning aims to use other related tasks to improve target task performance . Liu & Lane ( 2016b ) deal with traditional SLU piper-line network by jointly detecting intent and doing slot filling . Sogaard & Goldberg ( 2016 ) share parameters among various tasks , according to the low-level and high-level difference . Hershcovich et al . ( 2018 ) divide the representation network into task-specific and general which is shared during multi-task learning . Fan et al . ( 2017 ) and Herzig & Berant ( 2017 ) directly share the encoder or decoder neural layers ( model params ) through different semantic parsing tasks . In Kollar et al . ( 2018b ) , multi-task learning also mainly acts sharing the params of the network . 3 APPROACH . 3.1 DEFINITION OF CROSS-DOMAIN SKETCH . For human language expressions , especially task-oriented requests , there exist commonalities across sentences including linguistic syntax and phrase similarity . They can be seen with general sentence templates . Table 1 shows some examples . Since sentence templates are too many , we try to leverage these common regularities more abstractly . We extract these invariant commonalities which are implicit across domains , and call them as crossdomain sketch ( CDS ) in a canonical way . We define CDS in meaning representation language ( MRL ) format ( action-object-attribute ) and on two levels ( action-level and attribute-level ) . Action-level CDS means to acquire the same action for utterances from different domains while the attribute-level CDS means to extract more detailed information . See examples in Table 1 . Instead of extracting CDS from utterance directly , we try converting from logic form into CDS reversely , because it is easier to deal with structural logic form than utterance in natural language form . We analyze the dataset Snips and use a rule-based method to obtain CDS . We strip logic forms from domain-specific components and preserve domainindependent parts including general actions and attributes . We do some statistics on the dataset Snips ( Goo et al . ( 2018 ) ) used in this paper . We convert attributes [ object type , movie type , restaurant type ] into { object type } , [ object name , restaurant name , movie name ] into { object name } , [ year , timeRange ] into { time } , [ location name , current location ] into { object location } , [ country , city ] into { place } . All those attributes account for 55 % of all attributes which indicate the existence and feasibility of CDS . 3.2 MODEL . Figure 1 shows our overall network , which contains a two-level encoder-decoder . The General Network encodes utterance and decodes cross-domain sketch ( CDS ) . Since this process is domaingeneral , it can be done to all domains , which is a multi-task setup . The Detailed Network firstly encodes the CDS and the utterance , then it decodes the target result based on both utterance and CDS . This process is domain-dependent , so that it is a fine-tuning process in a specific domain . 3.2.1 PROBLEM DEFINITION . For an input utterance u = u1 , u2 , ... u|u| , and its middle result cross-domain sketch ( CDS ) c = c1 , c2 , ... c|c| , and its final result logic form y = y1 , y2 , ... y|y| , the conditional probability is : p ( y|u , c ) = |y|∏ t=1 p ( yt|y < t , u , c ) p ( c|u ) = |c|∏ t=1 p ( ct|c < t , u ) ( 1 ) where y < t = y1 , y2 , ... y|t−1| , and c < t = c1 , c2 , ... c|t−1| . 3.2.2 UTTERANCE ENCODER . The neural encoder of our model is similar to neural machine translation ( NMT ) model , which uses a bi-directional recurrent neural network . Firstly each word of utterance is mapped into a vector ut ∈ Rd via embedding layer and we get a word sequence u = ( u1 , ... , u|u| ) . Then we use a bi-directional recurrent neural network with long short-term memory units ( LSTM ) ( Hochreiter & Schmidhuber ( 1997 ) ) to learn the representation of word sequence . We generate forward hidden state −→ hut = fLSTM ( −−→ hut−1 , ut ) and backward state ←− hut = fLSTM ( ←−− hut−1 , ut ) . The t-th word will be hut = [ −→ hut , ←− hut ] . We construct two kinds of utterance encoders , general utterance encoder for General Network and specific utterance encoder for Detailed Network ( see in Figure 1 ) , so as to extract different information for different purposes . The general utterance encoder , meant to pay more attention to cross-domain commonalities of utterances , is used by all utterances from all domains . The specific utterance encoder , which is domain-dependent , belongs to one specific domain and is more sensitive to details . We call encoder outputs hugt from general utterance encoder and h us t from specific utterance encoder . When the two encoders share the same parameters that is hugt = h us t , we call it identical encoding and when they are not , we call it separate encoding , inspired by ( Sogaard & Goldberg ( 2016 ) ; Krishnamurthy et al . ( 2017 ) ; Liu et al . ( 2017 ) ; Abdou et al . ( 2018 ) ) which explore the sharing mechanisms of multi-task learning and propose some improvements .

Summary: This paper introduces “Cross domain schemas” (CDS) for semantic parsing of utterances made to a virtual assistant. CDS captures similarities in requests according to the underlying actions or attributes being discussed, regardless of the user’s high-level intent. Also introduced is a model which leverages CDS to improve semantic parsing of utterances to a meaning representation language (MRL). This model first parses an utterance to CDS, then uses an encoding of the CDS jointly with the utterance encoding to decode a meaning representation. By treating different intents as separate domains, the authors construct a multi-task learning setup for CDS and MRL parsing. Results are provided for the Snips dataset of virtual assistant queries.