Multi-Task Learning for Semantic Parsing with Cross-Domain Sketch

1 INTRODUCTION . Recently many natural language processing ( NLP ) tasks based on the neural network have shown promising results and gained much attention because these studies are purely data-driven without linguistic prior knowledge . Semantic parsing task which maps a natural language sentence into a machine-readable representation ( Fan et al . ( 2017 ) ) , as a particular translation task , can be treated as a sequence-to-sequence problem ( Dong & Lapata ( 2016 ) ) . Lately , a compositional graph-based semantic meaning representation language ( MRL ) has been introduced ( Kollar et al . ( 2018a ) ) , which converts utterance into logic form ( action-object-attribute ) , increasing the ability to represent complex requests . This work is based on MRL format for semantic parsing task . Semantic parsing highly depends on the amount of annotated data and it is hard to annotate the data in logic forms such as Alexa MRL . Several researchers have focused on the area of multi-task learning and transfer learning ( Hakkani-Tür et al . ( 2016 ) , Fan et al . ( 2017 ) , Kollar et al . ( 2018b ) ) with the observation that while these tasks differ in their domains and forms , the structure of language composition repeats across domains ( Herzig & Berant ( 2017 ) ) . Compared to the model trained on a single domain only , a multi-task model that shares information across domains can improve both performance and generalization . However , there is still a lack of interpretations why the multi-task learning setting works ( Ruder ( 2017 ) ) and what the tasks have shared . Some NLP studies around language modeling ( Le & Mikolov ( 2014 ) , Vaswani et al . ( 2017 ) , Devlin et al . ( 2018 ) ) indicate that implicit commonalities of the sentences including syntax and morphology exist and can share among domains , but these commonalities have not been fully discussed and quantified . To address this problem , in this work , compared to multi-task learning mentioned above which directly use neural networks to learn shared features in an implicit way , we try to define these cross-domain commonalities explicitly as cross-domain sketch ( CDS ) . E.g. , Search weather in 10 days in domain Weather and Find schedule for films at night in domain ScreeningEvent both have action SearchAction and Attribute time , so that they share a same MRL structure like SearchAction ( Type ( time @ ? ) ) , where Type indicates domain and ? indicates attribute value which is copying from the original utterance . We extract this domain general MRL structure as CDS . Inspired by the research of coarse-to-fine ( Dong & Lapata ( 2018 ) ) , we construct a two-level encoder-decoder by using CDS as a middle coarse layer . We firstly use General Network to get the CDS for every utterance in all domains . Then for a single specific domain , based on both utterance and extracted CDS , we decode the final target with advanced attention while CDS can be seen as adding some constraints to this process . The first utterance-CDS process can be regarded as a multi-task learning setup since it is suitable for all utterances across the domains . This work mainly introducing CDS using multi-task learning has some contributions listed below : 1 ) We make an assumption that there exist cross-domain commonalities including syntactic and phrasal similarity for utterances and extract these commonalities as cross-domain sketch ( CDS ) which for our knowledge is the first time . We then define CDS on two different levels ( action-level and attribute-level ) trying to seek the most appropriate definition of CDS . 2 ) We propose a general-to-detailed neural network by incorporating CDS as a middle coarse layer . CDS is not only a high-level extraction of commonalities across all the domains , but also a prior information for fine process helping the final decoding . 3 ) Since CDS is cross-domain , our first-level network General Network which encodes the utterance and decodes CDS can be seen as a multi-task learning setup , capturing the commonalities among utterances expressions from different domains which is exactly the goal of multi-task learning . 2 RELATED WORK . 2.1 SPOKEN LANGUAGE UNDERSTANDING . Traditional spoken language understanding ( SLU ) factors language understanding into domain classification , intent prediction , and slot filling , which proves to be effective in some domains ( Gupta et al . ( 2006 ) ) . Representations of SLU use pre-defined fixed and flat structures , which limit its expression skills like that it is hard to capture the similarity among utterances when the utterances are from different domains ( Kollar et al . ( 2018b ) ) . Due to SLU ’ s limited representation skills , meaning representation language ( MRL ) has been introduced which is a compositional graph-based semantic representation , increasing the ability to represent more complex requests ( Kollar et al . ( 2018a ) ) . There are several different logic forms including lambda-calculus expression ( Kwiatkowski et al . ( 2011 ) ) , SQL ( Zhong et al . ( 2018 ) ) , Alexa MRL ( Kollar et al . ( 2018a ) ) . Compared to fixed and flat SLU representations , MRL ( Kollar et al . ( 2018a ) ) based on a large-scale ontology , is much stronger in expression in several aspects like cross-domain and complex utterances . 2.2 SEQUENCE-TO-SEQUENCE FOR SEMANTIC PARSING . Mapping a natural language utterance into machine interpreted logic form ( such as MRL ) can be regarded as a special translation task , which is treated as a sequence-to-sequence problem ( Sutskever et al . ( 2014 ) ) . Then Bahdanau et al . ( 2015 ) and Luong et al . ( 2015 ) advance the sequence-tosequence network with attention mechanism learning the alignments between target and input words , making great progress in the performance . Malaviya et al . ( 2018 ) explore the attention mechanism with some improvements by replacing attention function with attention sparsity . Besides , to deal with the rare words , Gu et al . ( 2016 ) incorporate the copy mechanism into the encoder-decoder model by directly copying words from inputs . Lately , many researchers have been around improving sequence-to-sequence model itself , in interpreting the sentence syntax information . Eriguchi et al . ( 2016 ) encode the input sentence recursively in a bottom-up fashion . Wu et al . ( 2017 ) generate the target sequence and syntax tree through actions simultaneously . Another aspect which has caught much attention is constrained decoding . Krishnamurthy et al . ( 2017 ) and Post & Vilar ( 2018 ) add some constraints into decoding process , making it more controllable . Dong & Lapata ( 2016 ) use the recurrent network as encoder which proves effective in sequence representation , and respectively use the recurrent network as decoder and tree-decoder . Krishnamurthy et al . ( 2017 ) employ the grammar to constrain the decoding process . Dong & Lapata ( 2018 ) , believe utterance understanding is from high-level to low-level and by employing sketch , improve the performance . 2.3 MULTI-TASK LEARNING . For semantic parsing task especially in MRL format , it is expensive and time-consuming to annotate the data , and it is challenging to train semantic parsing neural models . Multi-task learning aims to use other related tasks to improve target task performance . Liu & Lane ( 2016b ) deal with traditional SLU piper-line network by jointly detecting intent and doing slot filling . Sogaard & Goldberg ( 2016 ) share parameters among various tasks , according to the low-level and high-level difference . Hershcovich et al . ( 2018 ) divide the representation network into task-specific and general which is shared during multi-task learning . Fan et al . ( 2017 ) and Herzig & Berant ( 2017 ) directly share the encoder or decoder neural layers ( model params ) through different semantic parsing tasks . In Kollar et al . ( 2018b ) , multi-task learning also mainly acts sharing the params of the network . 3 APPROACH . 3.1 DEFINITION OF CROSS-DOMAIN SKETCH . For human language expressions , especially task-oriented requests , there exist commonalities across sentences including linguistic syntax and phrase similarity . They can be seen with general sentence templates . Table 1 shows some examples . Since sentence templates are too many , we try to leverage these common regularities more abstractly . We extract these invariant commonalities which are implicit across domains , and call them as crossdomain sketch ( CDS ) in a canonical way . We define CDS in meaning representation language ( MRL ) format ( action-object-attribute ) and on two levels ( action-level and attribute-level ) . Action-level CDS means to acquire the same action for utterances from different domains while the attribute-level CDS means to extract more detailed information . See examples in Table 1 . Instead of extracting CDS from utterance directly , we try converting from logic form into CDS reversely , because it is easier to deal with structural logic form than utterance in natural language form . We analyze the dataset Snips and use a rule-based method to obtain CDS . We strip logic forms from domain-specific components and preserve domainindependent parts including general actions and attributes . We do some statistics on the dataset Snips ( Goo et al . ( 2018 ) ) used in this paper . We convert attributes [ object type , movie type , restaurant type ] into { object type } , [ object name , restaurant name , movie name ] into { object name } , [ year , timeRange ] into { time } , [ location name , current location ] into { object location } , [ country , city ] into { place } . All those attributes account for 55 % of all attributes which indicate the existence and feasibility of CDS . 3.2 MODEL . Figure 1 shows our overall network , which contains a two-level encoder-decoder . The General Network encodes utterance and decodes cross-domain sketch ( CDS ) . Since this process is domaingeneral , it can be done to all domains , which is a multi-task setup . The Detailed Network firstly encodes the CDS and the utterance , then it decodes the target result based on both utterance and CDS . This process is domain-dependent , so that it is a fine-tuning process in a specific domain . 3.2.1 PROBLEM DEFINITION . For an input utterance u = u1 , u2 , ... u|u| , and its middle result cross-domain sketch ( CDS ) c = c1 , c2 , ... c|c| , and its final result logic form y = y1 , y2 , ... y|y| , the conditional probability is : p ( y|u , c ) = |y|∏ t=1 p ( yt|y < t , u , c ) p ( c|u ) = |c|∏ t=1 p ( ct|c < t , u ) ( 1 ) where y < t = y1 , y2 , ... y|t−1| , and c < t = c1 , c2 , ... c|t−1| . 3.2.2 UTTERANCE ENCODER . The neural encoder of our model is similar to neural machine translation ( NMT ) model , which uses a bi-directional recurrent neural network . Firstly each word of utterance is mapped into a vector ut ∈ Rd via embedding layer and we get a word sequence u = ( u1 , ... , u|u| ) . Then we use a bi-directional recurrent neural network with long short-term memory units ( LSTM ) ( Hochreiter & Schmidhuber ( 1997 ) ) to learn the representation of word sequence . We generate forward hidden state −→ hut = fLSTM ( −−→ hut−1 , ut ) and backward state ←− hut = fLSTM ( ←−− hut−1 , ut ) . The t-th word will be hut = [ −→ hut , ←− hut ] . We construct two kinds of utterance encoders , general utterance encoder for General Network and specific utterance encoder for Detailed Network ( see in Figure 1 ) , so as to extract different information for different purposes . The general utterance encoder , meant to pay more attention to cross-domain commonalities of utterances , is used by all utterances from all domains . The specific utterance encoder , which is domain-dependent , belongs to one specific domain and is more sensitive to details . We call encoder outputs hugt from general utterance encoder and h us t from specific utterance encoder . When the two encoders share the same parameters that is hugt = h us t , we call it identical encoding and when they are not , we call it separate encoding , inspired by ( Sogaard & Goldberg ( 2016 ) ; Krishnamurthy et al . ( 2017 ) ; Liu et al . ( 2017 ) ; Abdou et al . ( 2018 ) ) which explore the sharing mechanisms of multi-task learning and propose some improvements .

Summary: This paper describes a two-stage encoder-decoder model for semantic parsing. The model first decodes a cross-domain schema (CDS) representation from the input utterance, then decodes the final logial form from both the utterance and CDS. The model outperforms other multitask Seq2Seq models on the Snips (Goo et al., 2018) dataset, but is still behind the traditional slot-filling models (Goo et al., 2018).