SlowGuess/NAACL_2021_OCR huggingface dataset:
Jie Wu
wuj8@uci.edu
Ian G. Harris
harris@ics.uci.edu
Hongzhi Zhao
hzzhao@bjtu.edu.cn
Spoken language understanding, usually including intent detection and slot filling, is a core component to build a spoken dialog system. Recent research shows promising results by jointly learning of those two tasks based on the fact that slot filling and intent detection are sharing semantic knowledge. Furthermore, attention mechanism boosts joint learning to achieve state-of-the-art results. However, current joint learning models ignore the following important facts: 1. Long-term slot context is not traced effectively, which is crucial for future slot filling. 2. Slot tagging and intent detection could be mutually rewarding, but bidirectional interaction between slot filling and intent detection remains seldom explored. In this paper, we propose a novel approach to model long-term slot context and to fully utilize the semantic correlation between slots and intents. We adopt a key-value memory network to model slot context dynamically and to track more important slot tags decoded before, which are then fed into our decoder for slot tagging. Furthermore, gated memory information is utilized to perform intent detection, mutually improving both tasks through global optimization. Experiments on benchmark ATIS and Snips datasets show that our model achieves state-of-the-art performance and outperforms other methods, especially for the slot filling task.
Task-oriented dialogue systems have attracted significant attention, which have been greatly advanced by deep learning techniques. Traditionally, these dialog systems have been built as a pipeline, with modules including spoken language understanding (SLU), dialog state tracking, action selection and language generation. Among these problems, SLU, including intention detection and slot filling (Tur and Mori, 2011), is a key yet challenging problem to parse users' utterances into se
Sentence	Flights	from	Irvine	to	Seattle
Intent	Flight
Slots	O	O	B-fromloc	O	B-toloc
Table 1: An example utterance annotated with its intent and semantic slots (IOB format).
mantic frames in order to capture a conversation's core meaning. Traditionally, intention detection is treated as a classification problem, whereas slot filling is usually defined as sequence labeling problem, where In-Out-Begin (IOB) format is applied for representing slot tags as illustrated in Table 1. Given an utterance, SLU determines users' intention and maps it into predefined semantic slots. The input is a sequence of words, and the output is a sequence of predefined slot IDs. A specific intent is assigned for the whole sentence.
In the traditional pipeline approach, intent detection and slot filling are implemented separately. However, separate modeling of those two tasks is insufficient to take full advantage of all supervised signals, as they share semantic knowledge. For example, if the intent of an utterance is "find_a_flight", it is more likely to contain slots "departure_city" and "arrival_city" rather than "restaurant_name". Another drawback of the pipeline method is that errors made in upper stream modules may propagate and be amplified in downstream components, which however could possibly be eased in joint model (Zhang and Wang, 2016).
Recently, joint model for intent detection and slot filling has been proposed and achieved promising results (Liu and Lane, 2016; Goo et al., 2018; Li et al., 2018). Though achieving promising performance, their models suffer from two major issues: 1) Modeling of slot context. Though the latent memory of RNNs can model history information, they are inherently unstable over long time sequences because the memories are the RNN hidden states. (Weston et al., 2014) observes that
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 797-806 June 6-11, 2021. ©2021 Association for Computational Linguistics
RNNs tend to focus more on short-term memories and forcefully compress historical records into one hidden state vector. Thus, simple RNNs cannot preserve long-term slot context of the conversation, which is crucial to future slot tagging. 2) Bi-directional interaction between slot filling and intent detection. The majority of joint modeling work has studied how to utilize intent information to improve slot filling performance. However, the beneficial impact of slot information on intent detection is mostly ignored. In fact, slots and intents are closely correlative, thus mutually reinforcing each other.
In this paper, we propose a new framework to jointly model intent detection and slot filling in order to achieve a deeper level of semantic modeling. Specifically, our model is distinguished from previous work primarily in two ways.
- Model slot context dynamically with Key-Value Memory Networks (KV-MNs). The majority of existing work use RNNs to track slot values mentioned in previous utterances. However, RNNs tend to focus more on short-term memories. We propose to use a memory network to model slot context information as external knowledge which is acting a global information to guide slot tagging. Instead of relying on the compressed vector in RNN, KV-MNs store different historical slot tag information separately in different memory slots, which enriches the representation capacity compared with RNNs. Furthermore, slot values mentioned in the utterance are dynamically tracked, which is beneficial for subsequent slot tagging at each timestamp. Lastly, slot-level attention can model more accurately the contribution of each word in an utterance to slot tagging.
- Model the mutual interaction between intent detection and slot filling. The fact that intent detection and slot filling are semantically related is well-observed and how to use intent information to boost slot filling is widely explored. However, slot filling is beneficial to intent detection as well, and these benefits are yet to be explored. We propose a gating mechanism between intents and slots based on KV-MNs in order to model the interaction between intent detection and slot filling.
Since intent detection can be treated as an utterance classification problem, different classification methods, such as support vector machines (SVM) and RNNs (Haffner et al., 2003; Sarikaya et al., 2011), have proposed to solve it. On the other hand, for slot filling, hidden markov models (HMM) and conditional random fields (CRF) (Lee et al., 1992; Ye-Yi Wang et al., 2005; Raymond and Riccardi, 2007) were used to solve slot filling problem. Later RNN based methods had become popular. For example, Yao et al. (2013); Mesnil et al. (2015) employed RNNs for sequence labeling in order to perform slot filling.
Alternatively, intent detection and slot filling can be done jointly to overcome the error propagation. Zhang and Wang (2016) first proposed joint work using RNNs for learning the correlation between intent and slots. Hakkani-Tur et al. (2016) adopted a RNN for slot filling and the last hidden state of the RNN was used to predict the utterance intent. Liu and Lane (2016) introduced an attention-based RNN encoder decoder model to jointly perform intent detection and slot filling. An attention weighted sum of all encoded hidden states was used to predict the utterance intent. All those models outperform the pipeline models via mutual enhancement between two tasks.
Most recently, some work tries to model the intent information for slot filling explicitly in the joint model. Goo et al. (2018); Li et al. (2018) proposed the gate mechanism to explore incorporating the intent information for slot filling. However, as the sequence becomes longer, it is risky to simply rely on the gate function to sequentially summarize and compress all slots and context information in a single vector (Cheng et al., 2016). Wang et al. (2018) proposed the bi-model to consider the cross-impact between the intent and slots and achieve state-of-the-art results. Zhang et al. (2018) proposed a hierarchical capsule neural network to model the hierarchical relationship among word, slot, and intent in an utterance. Niu et al. (2019) introduces a SF-ID network to establish the interrelated mechanism for slot filling and intent detection tasks. Compared with their work, our method explicitly models the long-term slot context knowledge which is beneficial to both slot filling and intent detection.
Memory network provides a principled approach for modeling long-range dependency which has advanced many NLP tasks such as machine transla
tion (Wang et al., 2016) and question answering (Sukhbaatar et al., 2015). The initial framework of memory networks was proposed by Weston et al. (2014). Following the idea, Sukhbaatar et al. (2015) proposed an end-to-end memory augmented model that significantly reduced the requirement of supervision during training. Key-value memory network (Miller et al., 2016) encoded prior knowledge by introducing a key memory structure which stored facts to address to the relevant memory value.
None of them is to model slot context information dynamically especially in single turn conversational systems. In this paper, we demonstrate how memory networks can be used to model long-term slot context knowledge and the interaction between intent detection and slot filling.
Memory networks show promising results on learning long-range dependency, but they are insensitive to represent temporal dependencies between memories (Wu et al., 2018). RNNs tend to be opposite. Thus, it makes sense for us to combine those networks together to model long-term slot context information. In this section, we present a specific key-value dynamic memory module to collect and remember slot clues in the dialog context. Then context memory is used to enhance the Encoder-Decoder based model to perform slot filling and intent detection.
Figure 1: Framework of the proposed model
As illustrated in Figure 1, our proposed model is composed of an Encoder-Decoder, and a Key-Value Memory Module including KEY-MEMORY, VALUE-MEMORY, a memory read unit, and a memory write unit. Given a single-turn dialog, the Encoder transforms a word in user utterances into a dense vector by using a shared self-attentive encoder. Then the memory network encodes long-term slot context information by incorporating historical slot tags through memory attention and WRITE operations of the memory network. The slot decoder integrates short-term hidden state of self-attention encoder and the long-term slot context generated by attentively reading the VALUE-MEMORY to generate slot tagging at each times-tamp. Later, intent decoder performs token level intent detection, which is seen as a coarse-grained intent detection result. Finally, a fine-grained intent detection is produced by gating memory modules. Both intent detection and slot filling are optimized simultaneously via a joint learning scheme.
3.1 Self-Attentive Encoder
Given an input utterance  $\mathbf{X} = (x_{1}, x_{2}, \ldots, x_{T})$  of  $T$  words, where each word is initially represented by a vector of dimension  $d$ , the BiLSTM (Hochreiter and Schmidhuber, 1997) is applied to learn representations of each word by reading the input utterance forward and backward to produce context sensitive hidden states  $\mathbf{H} = (h_{1}, h_{2}, \ldots, h_{T})$ :
$$
h _ {t} = \mathbf {B i L S T M} \left(x _ {t}, h _ {t - 1}\right) \tag {1}
$$
Then, we use self-attention mechanism to capture the contextual information for each token. We adopt the method proposed by (Vaswani et al., 2017), where we first map the matrix of input vectors  $\mathbf{X} \in \mathbb{R}^{T \times d}$  to queries  $(\mathbf{Q})$ , keys  $(\tilde{\mathbf{K}})$  and values  $(\tilde{\mathbf{V}})$  matrices by using different linear projections and the self-attention output  $\mathbf{C} \in \mathbb{R}^{T \times d_1}$  is:
$$
\mathbf {C} = \operatorname {s o f t m a x} \left(\frac {\mathbf {Q} \tilde {\mathbf {K}} ^ {\top}}{\sqrt {d _ {2}}}\right) \tilde {\mathbf {V}} \tag {2}
$$
where  $d_{1}$  and  $d_{2}$  represents self-attention dimension and keys'dimension. We concatenate the output of self-attention and BiLSTM as the final encoding representation as shown in Qin et al. (2019):
$$
\mathbf {E} = \mathbf {H} \oplus \mathbf {C} \tag {3}
$$
where  $\mathbf{E} = (e_1,\ldots ,e_T)\in \mathbb{R}^{T\times (d + d_1)}$  and  $\oplus$  is a concatenation operation.
3.2 Slot Decoder
Our slot decoder consists of two components: 1) the key-value memory-augmented attention model which generates slot context representation of users' utterance, and 2) the unidirectional LSTM decoder, which predicts the next slot tag step by step.
3.2.1 Dynamic Key Value Memory Network
To overcome the shortcomings of RNNs in capturing semantic clues over the long-term, we design a memory network that can preserve fine-grained semantic information of long-term slot context. We adopt a key-value memory network, which memorizes information by using a large array of external memory slots. The external memories enrich the representation capability compared with hidden vectors of RNNs and enable the KV-MNs to capture long-term data characteristics (Liu and Perez, 2017). We aim to incorporate the knowledge contained in the historical slot tags into the memory slots. The KV-MNs decompose slot semantics in an utterance into different slot categories and thus preserves more fine-grained information. In KV-MNs, a memory slot is represented by a key vector and an associated value vector.
- KEY-MEMORY: The KEY-MEMORY  $\mathbf{K} \in \mathbb{R}^{d_k \times n}$  learns latent correlation between utterance words and slot tags, where  $n$  is the number of memory slots and  $d_k$  is the dimension of each slot. Each column vector, that is,  $i$ -th key vector  $k_i \in \mathbb{R}^{d_k}$  is set to the  $i$ -th column of the KEY-MEMORY  $\mathbf{K}$ , which is shared by all conversation turns and fixed during the processing of word sequences.
- VALUE-MEMORY: Both the KEY-MEMORY and VALUE-MEMORY have the same number of memory slots. Each value memory vector stores the value of slot tag mentioned in the utterance. We form a value memory matrix  $\mathbf{V}_t \in \mathbb{R}^{d_v \times n}$  by combining all  $n$  value slots. Different from KEY-MEMORY K, VALUE-MEMORY V_t is word-specific and is continuously updated according to the input word sequence. During the conversation, the value of a new slot tag may be added into the VALUE-MEMORY, and an old value can be erased. In this way, we can adequately capture the slot context information on each mentioned slot. Two
types of operations, READ and WRITE, are designed to manipulate the value memories.
3.2.2 Memory-augmented Decoder
As shown in Figure 1, the decoder uses the aligned BiLSTM hidden state  $h_t$  as a query to address the KEY-MEMORY looking for an attention vector  $a_t$ , and attentively reads the VALUE-MEMORY to generate slot context representation  $c_t$ .
First, we use  $h_t$  to address the KEY-MEMORY to find an accurate attention vector  $a_t$ .
$$
a _ {t} = \mathbf {A d d r e s} \left(h _ {t}, \mathbf {K}\right) \tag {4}
$$
$a_{t}$  is subsequently used as the guidance for reading the VALUE-MEMORY  $\mathbf{V}_{t - 1}$  to get the slot context representation  $c_{t}$ .
$$
c _ {t} = \operatorname {R e a d} \left(a _ {t}, \mathbf {V} _ {t - 1}\right) \tag {5}
$$
$c_{t}$  works together with the aligned encoder hidden state  $e_t$  to generate the new decoder state at the decoding step  $t$ ,
$$
h _ {t} ^ {S} = \mathbf {L S T M} \left(h _ {t - 1} ^ {S}, y _ {t - 1} ^ {S}, e _ {t} \oplus c _ {t}\right) \tag {6}
$$
where  $h_{t-1}^{S}$  is the previous slot decoder state and  $y_{t-1}^{S}$  is the previous emitted slot lable distribution. After that, we use the slot decoder hidden state  $h_{t}^{S}$  to update  $\mathbf{V}_{t}$ :
$$
\mathbf {V} _ {t} = \mathbf {W r i t e} \left(h _ {t} ^ {S}, \mathbf {V} _ {t - 1}\right) \tag {7}
$$
Finally, the decoder state  $h_t^S$  is utilized for slot filling:
$$
y _ {t} ^ {S} = \operatorname {s o f t m a x} \left(\mathbf {W} _ {h} ^ {S} h _ {t} ^ {S}\right) \tag {8}
$$
$$
o _ {t} ^ {S} = \operatorname {a r g m a x} \left(y _ {t} ^ {S}\right) \tag {9}
$$
where  $\mathbf{W}_h^S$  are trainable parameters and  $o_t^S$  is the slot label of the word at timestamp  $t$  in the utterance.
3.3 Intent Detection Decoder
Different than most existing work where intent information is used to do slot filling, our framework is directly leveraging the explicit slot context information to help intent detection. Furthermore, a gated mechanism is used in order to effectively incorporate slot memory information into intent detection. By performing gated intent detection, there are two advantages:
1. Sharing slot context information with intent detection improves intent detection performance since those two tasks are related. Furthermore, a gating mechanism which combines the intent detection information and slot context retrieved from key-value memory, regulates the degree of enhancement of intent detection to prevent information overload.
2. Through shared key-value memory, the interaction between intent detection and slot filling can be effectively modeled and executed. Plus, by jointly training those two tasks, not only can intent detection performance be improved by slot context knowledge, but also slot filling is enhanced by minimizing intent detection objective function. In other words, by learning optimal parameters of shared key-value memory, slot filling and intent detection interact in a more effective and deeper way.
Intent Detection Decoder: For intent detection, we use another uni-directional LSTM as the intent detection network. At each decode step  $t$ , the decoder state  $h_t^I$  is generated by the previous decoder state  $h_{t-1}^I$ , the previous emitted intent label distribution  $y_{t-1}^I$  and the aligned encoder hidden  $e_t$ .
$$
h _ {t} ^ {I} = \mathbf {L S T M} \left(h _ {t - 1} ^ {I}, y _ {t - 1} ^ {I}, e _ {t}\right) \tag {10}
$$
Then the intent decoder state  $h_t^I$  together with the slot context  $c_t$  is utilized for final intent detection.
Gated Memory: We propose a gated mechanism to integrate slot context with intent detection. The gate regulates the degree of slot context information to feed into the intent detection task and prevent information from overloading. As shown in Figure 2, the gate  $\mathbf{G}$  is a trainable fully connected network with sigmoid activation.
Figure 2: Intent detection with gated memory
$$
h _ {t} ^ {\prime I} = g _ {t} \cdot h _ {t} ^ {I} + (1 - g _ {t}) \cdot c _ {t} \tag {11}
$$
where  $g_{t} =$  sigmoid  $\left(W_{t}[h_{t}^{I}\oplus c_{t}] + b_{t}\right)$ . Then, the output of gated decoder state  $h^{\prime}_{t}^{I}$  is utilized for intent detection:
$$
y _ {t} ^ {I} = \operatorname {s o f t m a x} \left(W _ {h} ^ {I} h _ {t} ^ {\prime I}\right) \tag {12}
$$
$$
o _ {t} ^ {I} = \operatorname {a r g m a x} \left(y _ {t} ^ {I}\right) \tag {13}
$$
where  $y_{t}^{I}$  is the intent output distribution of the  $t$ -th token in the utterance,  $o_{t}^{I}$  represents the intent lable of  $t$ -th token and  $W_{h}^{I}$  are trainable parameters of the model.
The final utterance result  $O^{I}$  is generated by voting from all token intent results as illustrated in Qin et al. (2019).
3.4 Memory Access Operation
In this section, we detail how to access key-value memory at the decoding time step  $t$ .
KEY-MEMORY Address:  $\mathbf{K} \in \mathbb{R}^{d_k \times n}$  denotes the KEY-MEMORY at decoding time step  $t$ . The addressed attention vector is given by
$$
a _ {t} = \mathbf {A d d r e s} \left(h _ {t}, \mathbf {K}\right) \tag {14}
$$
where  $a_{t} \in \mathbb{R}^{n}$  specifies the normalized weights assigned to the slots in  $\mathbf{K}$ , with  $j$ -th slot being  $\mathbf{k}_j$ . The attention weights  $a_{t,j}$  are calculated based on the correlation between  $h_t$  and  $\mathbf{k}_j$ :
$$
a _ {t, j} = \frac {\exp \left(e _ {t , j}\right)}{\sum_ {i = 1} ^ {n} \exp \left(e _ {t , i}\right)} \tag {15}
$$
where  $e_{t,j} = k_j^\top (\mathbf{W}_ah_t + b_a)$
VALUE-MEMORY Read:  $\mathbf{V}_t\in \mathbb{R}^{d_v\times n}$  denotes the VALUE-MEMORY at decoding time step  $t$  . The output of reading the value memory  $\mathbf{V}_t$  is given by
$$
c _ {t} = \sum_ {j = 1} ^ {n} a _ {t, j} \mathrm {v} _ {t, j} \tag {16}
$$
VALUE-MEMORY Write: Similar to the attentive writing operation of neural Turing machines (Graves et al., 2014), we define two types of operation for updating the VALUE-MEMORY: FORGET and ADD.
FORGET determines the content to be removed from memory slots. More specifically, the vector  $\mathbf{F}_t\in \mathbb{R}^{d_v}$  specifies the values to be forgotten or removed on each dimension in memory slots, which
is then assigned to each memory slot through normalized weights  $a_{t}$ . We use the slot decoder hidden state  $h_t^S$  to update  $\mathbf{V}_{t - 1}$ . Formally, the memory after FORGET operation is given by
$$
\tilde {\mathbf {v}} _ {t, i} = \mathbf {v} _ {t - 1, i} \left(\mathbf {1} - a _ {t, i} \cdot \mathbf {F} _ {t}\right), i = 1, 2, \dots , n \tag {17}
$$
-  $\mathbf{F}_t = \sigma(\mathbf{W}_F, h_t^S)$  is parameterized with  $\mathbf{W}_F \in \mathbb{R}^{d_v \times d_h}$ , and  $\delta$  stands for the Sigmoid activation function, and  $\mathbf{F}_t \in \mathbb{R}^{d_v}$ ;
-  $a_{t} \in \mathbb{R}^{n}$  specifies the normalized weights assigned to the key memory slots in  $\mathbf{K}$ , and  $a_{t,i}$  represents the weight associated with the  $i$ -th memory slot.
ADD decides how much current information should be written to the memory as the added content:
$$
\mathbf {v} _ {t, i} = \tilde {\mathbf {v}} _ {t, i} + a _ {t, i} \cdot \mathbf {A} _ {t}, i = 1, 2, \dots , n \tag {18}
$$
where  $\mathbf{A}_t = \sigma (\mathbf{W}_A,h_t^S)$  is parameterized with  $\mathbf{W}_A\in \mathbb{R}^{d_v\times d_h}$  and  $\mathbf{A}_t\in \mathbb{R}^{d_v}$ . By learning the parameters of FORGET and ADD layers, our model can automatically determine which signal to weaken or strengthen based on input utterance words.
The loss function for intent detection is  $\mathcal{L}_1$ , and that for slot filling is  $\mathcal{L}_2$ , which are defined as cross entropy:
$$
\mathcal {L} _ {1} \triangleq - \sum_ {j = 1} ^ {m} \sum_ {i = 1} ^ {n _ {I}} \hat {y} _ {j} ^ {I, i} \log \left(y _ {j} ^ {I, i}\right) \tag {19}
$$
$$
\mathcal {L} _ {2} \triangleq - \sum_ {j = 1} ^ {m} \sum_ {i = 1} ^ {n _ {S}} \hat {y} _ {j} ^ {S, i} \log \left(y _ {j} ^ {S, i}\right) \tag {20}
$$
where  $\hat{y}_j^{I,i}$  and  $\hat{y}_j^{S,i}$  are the gold intent label and gold slot label respectively,  $m$  is the number of words in a word sequence, and  $n_I$  and  $n_S$  are the number of intent label types and the number of slot tag types, respectively.
Finally the joint objective is formulated as weighted-sum of these two loss functions using hyper-parameters  $\alpha$  and  $\beta$ :
$$
\mathcal {L} _ {\theta} = \alpha \mathcal {L} _ {1} + \beta \mathcal {L} _ {2} \tag {21}
$$
Through joint training, the key-value memory shared by those two tasks can learn the shared representations and interactions between them, thus further promoting each other's performance and easing the error propagation compared with pipeline models.
To evaluate our proposed model, we conduct experiments on two widely used benchmark datasets, ATIS (Airline Travel Information System) and Snips. Both datasets used in our paper follow the same format and partition as in Goo et al. (2018). ATIS dataset (Hemphill et al., 1990) contains audio recordings of people making flight reservations. The training set has 4,478 utterances and the test set contains 893 utterances. We use another 500 utterances for the development set. There are 120 slot labels and 21 intent types in the training sets.