Transferring SLU Models in Novel Domains

Spoken language understanding ( SLU ) is a critical component in building dialogue systems . When building models for novel natural language domains , a major challenge is the lack of data in the new domains , no matter whether the data is annotated or not . Recognizing and annotating “ intent ” and “ slot ” of natural languages is a time-consuming process . Therefore , spoken language understanding in low resource domains remains a crucial problem to address . In this paper , we address this problem by proposing a transfer-learning method , whereby a SLU model is transferred to a novel but data-poor domain via a deep neural network framework . We also introduce meta-learning in our work to bridge the semantic relations between seen and unseen data , allowing new intents to be recognized and new slots to be filled with much lower new training effort . We show the performance improvement with extensive experimental results for spoken language understanding in low resource domains . We show that our method can also handle novel intent recognition and slot-filling tasks . Our methodology provides a feasible solution for alleviating data shortages in spoken language understanding . 1 INTRODUCTION . The recent surge of artificial intelligence motivates the technical and applicable exploration of novel human-computer interactions . Spoken dialogue systems are widely studied and used in various mobile devices . Well-known commercial applications driven by dialogue systems include intelligent personal assistants and intelligent robots for customer services . As more and more products and scenarios integrate dialogue systems in intelligent services , the ability for model adaptation in dialogue systems is critically needed . Many state-of-the art dialogue systems follow a learning pipeline that includes components such as spoken language understanding ( SLU ) , dialogue management ( DM ) and spoken language generation or retrieval ( SLG ) . Since both dialogue management and spoken language generation systems rely on knowledge or information learned from spoken language understanding , research and industry community have paid much attention to the SLU area . In this paper , we focus on the problem of data shortage for SLU in a new domain . Spoken language understanding typically defines and represents user utterances in terms of semantic frames comprised of domains , intents and slots ( Tur & Mori , 2011 ) . A spoken language understanding task involves classifying the domain , detecting the intent of user utterance and identifying token sequences corresponding to slot values in the semantic frame . A learning model is often designed to handle the above sub-tasks separately and sometimes simultaneously . Deep learning models are the cutting-edge techniques for achieving impressive performance in spoken language understanding compared with conventional machine learning models . However , deep neural-network models often require the collection and manual annotation of data , which is a time and labor intensive process . These problems pose a major challenge to building high-quality learning models in new domains . To address this problem , we propose to perform domain adaption of SLU models by transferring the SLU knowledge and model parameters learned in an auxiliary domain where a model is already built . A related problem is that only few examples of some output intent and slot classes are available during the training phrase in a new domain , because training sets for a new domain are small . To deal with this limitation , we propose a novel few shot learning model in a meta learning paradigm , we call it the Transfer Semantic Similarity Model ( TSSM ) . The TSSM achieves few-shot learning by exploiting the semantic embeddings of both seen and unseen intent and slots . In summary , the challenges of spoken language understanding include : ( 1 ) Expanding SLUs to novel domains is necessary as the development of artificial intelligent products extends to more areas in our society . But it is difficult to get sufficient data or spoken language sentence in novel domains . ( 2 ) Without sufficient labeled data , deep learning methods are difficult to train and thus it is hard to obtain satisfactory performance in a new domain . ( 3 ) Unseen intents and slots exist because small annotated training set might not cover all intents and slots . Thus , the generalization ability of the models is crucial for dealing with unseen classes . Our contributions are summarized as follows : • We design a novel and efficient SLU model , the TSSM . It incorporates transfer learning and meta learning for SLU tasks in low resource domains . • The proposed TSSM can improve model performance with only a small amount of training data in new domains and it can handle unseen classes . The experiments illustrate that our proposed methods outperform the state-of-the-art baselines in many experimental domains . • Our work can treat intent and slot detection tasks as a structured multi-objective optimization problem , which further improves the accuracy of learning tasks in new domains . 2 RELATED WORK . Spoken language understanding is usually defined as a supervised learning problem , involving conventional machine learning models ( Young , 2002 ; Hahn et al. , 2011 ; Wang et al. , 2005 ) and deep learning models ( Mesnil et al. , 2015 ; Kurata et al. , 2016 ; Sarikaya et al. , 2011 ) on massive amount of annotated training data . Although unsupervised learning and semi-supervised learning based approaches have been proposed as well ( Chen et al. , 2013 ; 2015b ; a ) , deep learning approaches were shown to outperform most others . Besides the requirement of a large amount of annotated data being available , the intents and slots are usually predefined . As a result , they are inflexible to expand to new domains . Some researchers have paid attention to alleviate this limitation . An example is the work by Korpusik et al . , which retrains models to cover new intents and associated slots while redesigning a semantic schema ( Korpusik et al. , 2014 ) . There have also been some attempts to learn shared latent semantic representations or model parameters for multiple domains via transfer learning . A survey for transfer learning with applications is Pan & Yang ( 2010 ) . In general , the goal of transfer learning is to improve performance in novel domains that only has small data . In these target domains , it is critical to reduce the model re-training efforts by bridging the knowledge from source tasks to the new target tasks . In natural language processing ( NLP ) , researchers have studied cross-domain NLP problems but have mainly focused in cross-lingual problem settings ( Mou et al. , 2016 ; Buys & Botha , 2016 ; Kim et al. , 2017a ; b ) . In Hakkani-Tr et al . ( 2016 ) , both multi-task and transfer learning approaches are proposed to learn shared implicit feature representation across various domains for spoken language understanding . Researchers in Amazon have shown in a recent study that DNN-based natural language engines can re-use the existing models through transfer learning ( Goyal et al. , 2018 ) . When the training data is missing some task labels , one can exploit meta learning to apply few-shot learning to help with the knowledge transfer Fei-Fei et al . ( 2006 ) ; Palatucci et al . ( 2009 ) ; Vinyals et al . ( 2016 ) ; Ravi & Larochelle ( 2018 ) . For example , Vinyals et al . ( 2016 ) learns a network that maps a small amount of labeled support set and unlabeled examples to its labels , saving the effort in fine-tuning to new class types . Besides image object-recognition tasks , in SLU some experiments have been conducted . The objective of Ferreira et al . ( 2015 ) is to predict the semantic tag sequences of a user query without using any target-domain user utterances and thus in-context semantic tags . The research builds a statistical model to predict the SLU for unseen data in Yazdani & Henderson ( 2015 ) ; Chen et al . ( 2016 ) . An action-matching algorithm is proposed by Zhao & Eskénazi ( 2018 ) to learn a cross-domain embedding space that models the semantics of dialog responses , which , in turn , lets a neural dialog generation model generalize to new domains . Recent publication by Google ( Bapna et al. , 2017 ) explores semi-supervised slot-filling based on deep learning that can use the slot labels without the need for any labeled or unlabeled data in domain examples or the need for any explicit schema alignment , to quickly bootstrap the model in a new domain . 3 OUR PROPOSED APPROACH FOR MODEL TRANSFER . In this section , we introduce our transfer learning approach to address the cross-domain SLU problem . We consider a source domain Ds and a target domain Dt , where our goal is to learn a model in the target domain . Typically , we have a large annotated dataset for Ds , while only a few labeled data for Dt . The tasks in both Ds and Dt are of the same types , namely they are both slot filling and intent detection tasks . However , Ds and Dt have different intent label spaces Ts and Tt , because the target domain might have new intent values ( slot faces the same situation ) . The number of possible intents and slots grows rapidly as we get into new domains with new intents and slots . One natural approach in solving the learning problem is to train one binary classifier for each possible label , and decide whether or not to include the classification model in the output . However , this would require training a large number of classifiers . It also would be impossible to generalize to target domains that include intents and slots that do not show up in the training set , since there will not be any parameter sharing among the classifiers . Instead of encoding slot names and intent names as discrete features in the target domain , we encode them using a continuous representation . To be specific , we learn a representation of slot names , slot values and intent names from their constituent words using semantic networks . We then check to see if these representations match the representations of utterances . The architecture of our model is illustrated in Fig . 1 . The model includes the following major components . ( 1 ) a word hashing layer to convert one-hot word representations into embedding vectors , ( 2 ) a bi-directional Long Short Term Memory networks ( LSTMs ) ( Hochreiter & Schmidhuber , 1997 ) layer to capture features from word embeddings , ( 3 ) a intent specific bi-LSTMs layer combined with an semantic network to detect intent , ( 4 ) a slot specific bi-LSTMs layer powered with an attention component and semantic network to conduct slot filling . We first pretrain the model in a source domain with adequate annotated data . We then wish to allow the model to extend to new slot names , slot values and intent names in the target domain with little annotated data . Let w = ( w0 , w1 , w2 , . . . , wT+1 ) represent the input word sequence with w0 and wT+1 being the beginning-of-sequence ( 〈bos〉 ) and end-of-sequence ( 〈eos〉 ) tokens , respectively . T is the number of words . Let s = ( s0 , s1 , . . . , sT ) be the slot label sequence , where s0 is a padded slot label that maps to the beginning-of-sequence token 〈bos〉 and si ∈ S. Let τ ∈ T be the intent class . S and T are the slot and intent label space , respectively . Let sn = ( sn1 , sn2 , . . . , snm ) represent the m slot names . sni = ( w1 , . . . , wmi ) is a vector of the words for the i-th slot name , where mi is its number of words . Each slot si has a list of slot values svi = ( svi,1 , svi,2 , . . . , svi , ni ) , where ni is the number of slot values for the i-th slot . A slot value is allowed to be a phrase like ‘ fried chicken ’ for slot ’ Food ’ . Let svi , j = ( w1 , . . . , wni , j ) be the words of the j-th slot value for i-th slot and ni , j is the number of words for this slot value .

Summary: In this paper, an efficient SLU model, called as TSSM, is proposed to tackle the problem of insufficient training data for the task of spoken language understanding. TSSM considers the intent and slot detection as a unified multi-objective optimization problem which is addressed by a meta-learning scheme. The model is pre-trained on a large dataset and then fine-tuned on a small target dataset. Thus, the proposed TSSM can improve the model performance on a small datatset in new domains.