# Elastic CRFs for Open-Ontology Slot Filling
## 1 Introduction  
Slot filling [1, 2] is a crucial component in task-oriented dialog systems and parses (user) utterances into semantic concepts in terms of a set of named entities called slots. The example in Figure 1 contains the slots time and movie. In parsing, some span in the utterance is identified as the slot value for some slot; e.g., here, _\"6 pm\"_ is marked as the slot time. An ontology, which describes the scope of semantics that the dialog system can process, is defined by the collection of slots and the values that each slot can take. A widely used practice for slot filling is to introduce IOB tags [3] and assign a label to each token in the utterance. A label, e.g., B-time, is a combination of the slot name and one of the IOB tags.These labels are then used to identify the values for different slots from the utterance. In this manner, slot filling is treated as a sequence labeling task, as illustrated in Figure 1, for which the two dominant classes of methods are based on recurrent neural networks (RNNs) [1] and conditional random fields (CRFs) [4], respectively. This practice has been widely employed for slot filling [2, 5] and many other similar sequence labeling problems [6]. However, this practice suffers from two drawbacks. First, currently, most slot-filling methods are unable to predict new labels for unseen slots. The ontology is usually pre-defined and fixed. It is difficult to accommodate new semantic concepts (slots) in slot filling. However, users may often add new semantic concepts in a domain and dialog systems are expected to work across an increasingly wide range of domains. Thus, it is highly desirable for slot-filling models to be able to handle new slots, whether in-domain or cross-domain, with the least expentraining on a certain domain. In this paper, we are interested in developing such open-ontology slot filling, which means that the collection of slots and values is open-ended for slot filling. Second, in current slot-filling models [5, 7], slot labels are generally encoded as one-hot vectors. However, slot labels are not merely discrete classes. There are natural language descriptions for each slot, e.g., the description \"number of people\" for the slot #people. This one-hot encoding ignores the semantic meanings and relations for slots, which are implicit in their natural language descriptions and useful for slot filling. There are prior efforts to address the above two drawbacks. The difficulty of transferring between domains could be partly alleviated with multi-task learning [8, 9, 10], by performing joint learning on multiple domains. Practically, varying only the last output layer for different domains and sharing the parameters of the rest layers has shown to be a successful approach [11]. In this approach, the slot-filling model can leverage all available multi-domain data and transfer them to handle those slots with sparse training data. However, basically, this multi-task learning approach is unable to predict labels for zero-shot slots (namely those slots that are unseen in training data and whose values are unknown). It can be seen that this difficulty is also related to the drawback of one-hot encoding slot labels, which hinders the exploitation of semantic relations and shared statistical properties between different slots. A recent work [12] proposes utilizing slot label descriptions towards zero-shot slot filling by introducing slot encodings from natural language descriptions. Basically, they use RNN-based sequence labeling, taking the slot encoding vector as an additional conditional input and outputting the IOB tags in each position. Sequence labeling is carried out independently for all slots. Though yielding promising results, there are two shortcomings. First, independent sequence labeling may make conflicting predictions. Second, interactions between slots are ignored in sequence labeling. CRFs have been shown to be one of the most successful approaches for sequence labeling, especially for capturing the interactions between labels.