From General to Specific: Leveraging Named Entity Recognition for Slot Filling in Conversational Language Understanding
Authors: Samuel Louvan, Bernardo Magnini (Università di Trento, Fondazione Bruno Kessler, 2018)
Topic: Chatbots and Dialogue Systems

Abstract
Slot ﬁlling techniques are often English. Slot ﬁlling techniques are often adopted in language understanding components for task-oriented dialogue systems. In recent approaches, neural models for slot ﬁlling are trained on domainspeciﬁc datasets, making it difﬁcult porting to similar domains when few or no training data are available. In this paper we use multi-task learning to leverage general knowledge of a task, namely Named Entity Recognition (NER), to improve slot ﬁlling performance on a semantically similar domain-speciﬁc task. Our experiments show that, for some datasets, transfer learning from NER can achieve competitive performance compared with the state-of-the-art and can also help slot ﬁlling in low resource scenarios.

Introduction
In dialogue systems, semantic information of an utterance is generally represented with a semantic frame, a data structure consisting of a domain, an intent, and a number of slots (Tur, 2011). For example, given the utterance “I’d like a United Airlines ﬂight on Wednesday from San Francisco to Boston”, the domain would be ﬂight, the intent is booking, and the slot ﬁllers are United Airlines (for the slot airline name), Wednesday (booking time), San Francisco (origin), and Boston (destination). Automatically extracting this information involves domain identiﬁcation, intent classiﬁcation, and slot ﬁlling, which is the focus of our work. Slots are usually domain speciﬁc as they are predeﬁned for each domain. For instance, in the ﬂight domain the slots might be airline name, booking time, and airport name, while in the bus domain the slots might be pickup time, bus name, and travel duration. Recent successful approaches related to slot ﬁlling tasks (Wang et al., 2018; Liu and Lane, 2017a; Goo et al., 2018) are based on variants of recurrent neural network architecture. In general there are two ways of approaching the task: (i) by training a single model for each domain; or (ii) by performing domain adaptation, which results in a model that learns better feature representations across domains. All these approaches directly train the models on domain-speciﬁc slot ﬁlling datasets. In our work, instead of using a domain-speciﬁc slot ﬁlling dataset, which can be expensive to obtain being task speciﬁc, we propose to leverage knowledge gained from a more “general”, but semantically related, task, referred as the auxiliary task, and then transfer the learned knowledge to the more speciﬁc task, namely slot ﬁlling, referred as the target task, through transfer learning. In the literature, the term transfer learning can be used in different ways. We follow the deﬁnition from (Mou et al., 2016), in which transfer learning is viewed as a paradigm which enables a model to use knowledge from auxiliary tasks to help the target task. There are several ways to train this model: we can directly use the trained parameters of the auxiliary tasks to initialize the parameters in the target task (pre-train & ﬁne-tuning), or train a model of auxiliary and target tasks simultaneously, where some parameters are shared (multitask learning). We propose to train a slot ﬁlling model jointly with Named Entity Recognition (NER) as an auxiliary task through multi-task learning (Caruana, 1997). Recent studies have shown the potential of multi-task learning in NLP models. For example, (Mou et al., 2016) empirically evaluates transfer learning in sentence and question classiﬁcation tasks. (Yang et al., 2017) proposes an approach for transfer learning in sequence tagging tasks. NER is chosen as the auxiliary task for several reasons. First, named entities frequently occur as slot values in several domains, which make them a relevant general knowledge to exploit. The same NER type can refer to different slots in the same utterance. On the previous utterance example, the NER labels are LOC for both San Francisco and Boston, and ORG for United Airlines. Second, state-of-the-art performance of NER (Lample et al., 2016; Ma and Hovy, 2016) is relatively high, therefore we expect that the transferred feature representation can be useful for slot ﬁlling tasks. Third, large annotated NER corpora are easier to obtain compared to domain-speciﬁc slot ﬁlling datasets. The contributions of this work are as follows: we investigate the effectiveness of leveraging Named Entity Recognition as an auxiliary task to learn general knowledge, and transfer this knowledge to slot ﬁlling as the target task in a multi-task learning setting. To our knowledge, there is no reported work that uses NER transfer learning for slot ﬁlling in conversational language understanding. Our experiments show that for some datasets multi-task learning achieves better overall performance compared to previous published results, and performs better in some low resource scenarios.

Conclusion
In this work we train a slot ﬁlling domain-speciﬁc model adding NER information, under the assumption that NER introduces useful “general” labels, and that it is cheaper to obtain compared to task speciﬁc slot ﬁlling datasets. We use multitask learning to leverage the learned knowledge from NER to slot ﬁlling task. Our experiments show evidence that we can achieve comparable or better performance against the state-of-the-art approaches and against single task learning, both in full training data and low resource scenarios. In the future, we are interested in working on datasets in Italian and explore more sophisticated multitask learning strategies.