# Multitask Learning with Knowledge Base for Joint Intent Detection and Slot Filling
## 3 Multitask Learning with Knowledge Base for Joint Slot-Filling and Intent-Detection
### Development of the Intent-Detection and Slot-Filling Model  
The two Bi-LSTM models extract their respective characteristics for each task based on the shared parameter representations \(h^{(shared)}\). As shown in Figure 1, the output of the shared presentation layer flows in two parts. One part is the Bi-LSTM model with the attention mechanism that completes the intent detection task, and the other part achieves the slot filling task. Two training datasets \(Data_{intent}\) and \(Data_{slot}\) are constructed with the intent tag and the slot tag, respectively. We establish the Bi-LSTM models with the attention mechanism for the intent-detection and slot-filling tasks, respectively. The structure of this part is shown in Figure 2. Accordingly, the intent detection that predicts output \(y^{l}\) and the slot filling that predicts output \(y^{S}_{i}\) are obtained.  
Taking the text sequence \(x\) and \(h^{(shared)}\) as the input, the corresponding hidden layer output \(h=(h_{0},\cdots,h_{T})\) is obtained, and the attention mechanism is introduced in the hidden layer to calculate the probability distribution of attention \(att\). \(att_{i,j}\) is calculated as follows:  
\[att_{i,j}=\frac{exp\big{(}e_{i,j}\big{)}}{\sum_{k=1}^{T}exp(e_{i,k})}, \tag{1}\]  
\[e_{i,j}=f(h_{i-1},h_{i}). \tag{2}\]  
Figure 2: The structure of the intent-detection and slot-filling model. The O denotes the “none” type and B-X denotes that the segment in which this token resides is of type X and that this token is at the beginning of the segment.  
After obtaining the probability distribution of attention at any time, the feature vector \(c\) that contains the text information is calculated as follows:  
\[c^{I}=\sum_{j=0}^{T}att_{i,j}h_{j}. \tag{3}\]  
The last part is the output layer. The softmax function is applied to the representations using a linear transformation to obtain the distribution \(y^{I}\) over the intent labels:  
\[y^{I}=softmax(\textbf{W}_{s}h_{T}+\textbf{b}_{i}), \tag{4}\]  
where \(\textbf{W}_{s}\) and \(\textbf{b}_{i}\) is the weight matrix, and b represents the offset vector.  
The slot-filling maps the text sequence \(x\) to its corresponding slot sequence label \(y_{i}^{S}\). The Bi-LSTM generates the bidirectional hidden state \(h_{i}\) at time \(t\), which is defined as the concatenation of the forward hidden state and backward hidden state. For each hidden state \(h_{i}\), the slot context vector \(c_{i}^{s}\) is calculated as the weighted sums of the Bi-LSTM's hidden states \(h_{1},\cdots,h_{T}\) using the learned attention weights \(\alpha\). The slot context vector \(c_{i}^{s}\) can also be calculated in the same way as \(c^{I}\), and the slot label of the \(i\)-th word is expressed as follows:  
\[y_{i}^{S}=softmax(\textbf{W}_{s}h_{i}+\textbf{b}_{i}). \tag{5}\]  
During the training process, the results of the intent-detection and slot-filling model are used for modeling the slot-intent relationships:  
\[r=\sum\eta log\Big{(}\textbf{W}_{1}c^{I}+\textbf{W}_{2}c_{i}^{s}\Big{)}^{2}. \tag{6}\]  
This parameter updates \(y^{I}\) and \(y_{i}^{S}\) to influence the output of the intent-detection and slot-filling modules, where Equations (4) and (5) are reformed as (7) and (8) respectively:  
\[y^{I}=softmax(\textbf{W}_{s}h_{T}r+\textbf{b}_{i}), \tag{7}\]  
\[y_{i}^{S}=softmax(\textbf{W}_{s}h_{i}r+\textbf{b}_{i}). \tag{8}\]