## 3 Methodology
### Stacked BiLSTM for Feature Extraction  
To further learn better contextual and semantic features we deploy SBL on the top of BERT's last hidden state. As stated earlier, SBL is the extension of LSTM stacked with multiple layers of neural networks to learn the features from dialogue context deeply. It has been perceived that multiple layers of LSTM reduce the validation loss more than single layer of LSTM to enhance the accuracies of DST (Section 5.3). For better understanding of multiple layers of Stacked BiLSTM, we will investigate the intuition of LSTM. LSTM is used to solve the long-range dependency of the sequence and vanishing the gradient problem [72]. The vanishing gradient problem occurs when the gradients are back propagated through the network then the network can vastly rot or grow [73]. For example, when multiple layers using the activation function are added to a network then the gradients of the loss function approach toward zero, making its training a challenging task. The purpose of LSTM is to add, delete or update dialogue context and candidate information in cells organized by three gating mechanisms. Gating mechanisms at time step \(s\) renowned as input gate \(x_{s}\), forget gate \(y_{s}\) and output gate \(z_{s}\). These gating mechanisms with forward and backward layers control the flow of information for reading, writing, and updating in the last hidden states. The first step of the LSTM block is to decide which features from dialogue context have to be excluded from the cell status, equipped by forget gate as shown in Equation (2). \[y_{s}=\sigma(V^{f}I_{s}+h_{s-1}P^{f}+b_{f}) \tag{2}\]  
Here, \(I_{s}\) is the input \(V^{f}\) and \(P^{f}\) are the weight matrices, \(b_{f}\) bias vector parameters, \(h_{s-1}\) is the hidden state of the previous state at the time step \(s\). Then the next step for the LSTM block is to decide what batch of new features from dialogue context and candidates has to be stored in the cell state calculated by the input gate \(x_{s}\) about the corresponding values update and tanh layer that creates a vector of a new candidate \(\hat{C}_{s}\) as shown in Equations (3) and (4), respectively. Then, the status of the previously stored utterances and values are updated and calculated by element-wise multiplication \(\times\) between its inputs as shown in Equation (5). \[x_{s}=\sigma(V^{x}I_{s}+h_{s-1}P^{x}+b_{x}) \tag{3}\]  
\[\hat{C}_{s}=\sigma(V^{C}I_{s}+h_{s-1}P^{C}+b_{c}) \tag{4}\]  
\[C_{s}=(C_{s-1}\times x_{s}\times y_{s}\times\hat{C}_{s}) \tag{5}\]  
Lastly, the output gate \(z_{s}\) decides what output from which part of the cell status must be generated. This output is the updated version and mathematically represented in Equation (6). At each turn, if the new corresponding value against slot occurred during the current utterance then the input gate extracts the new value. Afterwards, the previous state is forgotten by forget gate and new state is updated by \(C_{s}\) to produce hidden state as shown in Equation (7). \[z_{s}=\sigma(V^{z}+h_{s-1}P^{z}+b_{z} \tag{6}\]  
\[h_{s}=z_{s}\times tanh(C_{s}) \tag{7}\]  
To enhance the performance of typical LSTM, we have implemented forward and backward layers of LSTM to encode past and future information of dialogue context and candidate. The input \(I_{t}\), which holds the last hidden state of BERT, is fed to the first BiLSTMblock with the previous hidden state \(h_{t-1}^{(1)}\) in the Stacked BiLSTM. We have used two stacked layers of BiLSTM. The hidden state at the time step \(s\) is calculated as mentioned earlier. Later, it moves to the next time step while also shifting up to the second block of Stacked BiLSTM. The forward and backward stacked layers of LSTM read all the words in the last hidden state \(U_{h}=U_{h}^{L}(L=12)\) generated by BERT to output the actual output \(r_{s}\), which contains the whole hidden states. Meanwhile, the second output for hidden features of the LSTM \(H_{s}\) is generated.