## 3 Model Description
### Bi-LSTM Encoding with Attention  
Having solved the exploding and vanishing gradient problems of RNN, LSTM has enjoyed a lengthy time-line of being the basic model widely used in NLP tasks due to its capability to capture long-term textual dependencies [19]. Moreover, the enhanced bi-directional variant of the traditional LSTM which comprises of 2 hidden layers in opposing directions enables it to connect past and future input features to the same output in specified time steps, thus, the Bi-LSTM is able to learn fast. Meanwhile, comparing Bi-LSTM to the bi-directional gated recurrent unit (Bi-GRU) [20; 21], the study [20] showed that the latter is equally fast; however, we justify our choice of adopting the Bi-LSTM in the mDNA model based on the analysis in [22] that the LSTM is a better option than GRU for performing tasks where deep context understanding is the major consideration of our study.  
In this section, we apply the two input sequences of memory and response, \(M\) and \(R\), into the Bi-LSTM encoder to represent the tokens of the contextual vectors, and we denote the vectors of the hidden state as \(m^{s}\) and \(r^{s}\):  
Figure 3: Storage system concept of the memory unit.  
\[m_{i}^{s} =\text{BiLSTM}(M,i), \tag{1}\] \[r_{j}^{s} =\text{BiLSTM}(R,j), \tag{2}\]  
where \(i\) and \(j\) represent the \(i\)-th context in the utterance, and the \(j\)-th context in the response, respectively.  
To determine whether a selected response candidate is appropriate and correlated with the context of the utterance, modeling the relationship between the utterance and the response is an essential step. For example, appropriate responses consider contextual keyword vectors, which can be obtained by modeling the semantic relationship. To this regard, being that not all words contribute equally to the context representation of an utterance, attention is required to ensure that important information are collected for matching composition. In this case, we use a multi-head attention mechanism to align the utterance contexts to the response candidates, and then calculate the semantic relationship at the utterance level. We also applied a soft alignment layer to calculate the attention weights as follows:  
\[e_{ij}=(m_{i}^{s})^{T}*r_{j}^{s}, \tag{3}\]  
where \(e\in R^{wsn}\), and \(T\) is the length of the sequence. For the hidden state of utterance context, i.e., \(m_{i}^{s}\) (already encoding the utterance and its contextual meaning), the relevant semantics in the response options are identified and composed using \(e_{ij}\) as a vector \(m_{i}^{d}\), more specifically as shown in Equation (4).  
\[\alpha_{ij}=\exp(e_{ij})/\sum_{k=1}^{n}\exp(e_{ik}),m_{i}^{d}=\sum_{j=1}^{n} \alpha_{ij}r_{j}^{s}, \tag{4}\]  
\[\beta_{ij}=\exp(e_{ij})/\sum_{k=1}^{w}\exp(e_{kj}),r_{j}^{d}=\sum_{i=1}^{w} \beta_{ij}m_{i}^{s}, \tag{5}\]  
where \(\alpha\in R^{w}*n\) and \(\beta\in R^{w}*n\) are the normalized attention weight matrices. The same process is also performed for each utterance and response matching in Equation (5).  
Hence, we model the utterance level semantic relationship between the aligned utterance pairs, i.e., \(<m_{i}^{s},m_{i}^{d}>\) and \(<r_{i}^{s},r_{i}^{d}>\). The equation is further expressed as follows:  
\[m_{i}^{w}=P([m_{i}^{s};m_{i}^{d};m_{i}^{s}-v_{i}^{d};m_{i}^{s}\odot m_{i}^{d}]), \tag{6}\]  
\[r_{i}^{w}=P([r_{i}^{s};r_{i}^{d};r_{i}^{s}-r_{i}^{d};r_{i}^{s}\odot r_{i}^{d}]), \tag{7}\]  
where a matching layer can be used to model some high-order interaction between the vectors \(m_{i}^{w}\) and \(r_{i}^{w}\) for the utterance and response, respectively. \(P\) is a 1-layer multi-layer perception (MLP) with a rectified linear unit (ReLU) activation.