# Multitask Learning with Knowledge Base for Joint Intent Detection and Slot Filling
## 3 Multitask Learning with Knowledge Base for Joint Slot-Filling and Intent-Detection
### Optimization of Intent Detection and Slot Filling Model  
The loss function is used to characterize the error between the output value of the neural network and the true value. The influence of the loss function on the adjustable parameters of the neural network cannot be ignored. If the loss function is not used properly, the parameters of the neural network will not be satisfactory, even if the training is performed many times.  
In our joint model, the cross entropy can be used as a loss function in the back propagation of neural networks and plays a role in the network parameters' update.  
Particularly, the loss function \(Loss_{intent}\) is defined as the cross-entropy of the predicted output \(y^{I}\) and the true intent \(y^{intent}\).  
\[Loss_{intent}=-\sum_{i=1}^{T}y^{intent}log\Big{(}y^{I}\Big{)}. \tag{12}\]  
To ensure that the result of each iteration is close to the real label, the loss function of slot filling \(Loss_{slot}\) is calculated as the average cross-entropy of the slot-filling model's predicted output \(y^{S}_{i}\) and the real slot sequence.  
\[Loss_{slot}=\frac{1}{T}\sum_{i=1}^{T}L\Big{(}y^{S}_{i},y^{slot}_{i}\Big{)},L \Big{(}y^{S}_{i},y^{slot}_{i}\Big{)}=-\sum_{i=1}^{M}y^{slot}_{i}log(y^{S}_{i}). \tag{13}\]  
In Equation (13), \(y^{S}_{i}\) is the sequence label of the i-th iteration, \(y^{slot}_{i}\) is the true label, \(T\) is the total number of iterations, \(M\) is the length of sentence, and \(L\) is the cross-entropy.  
MTL needs to consider the proportion of the loss function of each task. Different proportions determine the importance of the shared information that is provided by different tasks. In the joint training process, the final loss function is the summation of the losses that were obtained by completing different tasks. In this paper, the weighted-loss function is constructed based on taking the proportions of the intent-detection model and the slot-filling model as the weight parameters. The loss function in our joint model is shown in Equation (14).  
\[Loss=\alpha Loss_{intent}+\beta Loss_{slot}s.t.\ \alpha+\beta=1 \tag{14}\]  
In Equation (14), the function \(Loss\) is the total loss of the joint model, and \(\alpha\) and \(\beta\) are the weight coefficients of the preset intent-detection task and the slot-filling task, respectively.  
The gradient descent algorithm is conducive to quickly finding the optimal solution. The setup of \(\alpha\) is established by using the weighted self-learning method based on the gradient descent algorithm, with the calculation steps as follows:  
\[\begin{array}{l}\frac{\partial Loss}{\partial\alpha}=\sum\limits_{X}\Big{(} \frac{t}{f(z)}-\frac{1-t}{1-f(z)}\Big{)}\frac{\partial f}{\partial\alpha}\\ =\sum\limits_{x}\Big{(}\frac{t}{f(z)}-\frac{1-t}{1-f(z)}\Big{)}f^{\prime}(z)x \\ =\sum\limits_{x}\frac{x^{\prime}(z)}{f(z)(1-f(z))}(f(z)-t)\\ =\sum\limits_{x}x(f(z)-t)\end{array} \tag{15}\]In Equation (15), \(f(z)\) represents the output value of the model, \(t\) is the true value of the sample, and \((f(z)-t)\) is the error between the output value and the true value \(t\) of the sample. Therefore, when the value of \((f(z)-t)\) is larger, the error is larger, the gradient value is larger, the weight parameter \(\alpha\) is adjusted faster, and the training speed is also faster.  
With the gradient calculation being performed on the weight parameter, the value \(\alpha\) is iteratively updated using Equation (16).  
\[\alpha\leftarrow\alpha d\frac{\partial Loss}{\partial\alpha}. \tag{16}\]  
In Equation (16), \(d\) is expressed as the learning rate of the gradient step.  
When the monotonicity of loss cannot be maintained, the iteration is stopped and the value \(\alpha\) is obtained.  
Finally, we use Adam, an algorithm for the first-order gradient-based optimization of stochastic objective functions, as an optimization method to optimize our model.