Question: What QA models were used?
Evidence: The input of our model are the words in the input text $x[1], ... , x[n]$ and query $q[1], ... , q[n]$ . We concatenate pre-trained word embeddings from GloVe BIBREF40 and character embeddings trained by CharCNN BIBREF41 to represent input words. The $2d$ -dimension embedding vectors of input text $x_1, ... , x_n$ and query $q_1, ... , q_n$ are then fed into a Highway Layer BIBREF42 to improve the capability of word embeddings and character embeddings as
$$\begin{split} g_t &= {\rm sigmoid}(W_gx_t+b_g) \\ s_t &= {\rm relu } (W_xx_t+b_x) \\ u_t &= g_t \odot s_t + (1 - g_t) \odot x_t~. \end{split}$$ (Eq. 18)
Here $W_g, W_x \in \mathbb {R}^{d \times 2d}$ and $b_g, b_x \in \mathbb {R}^d$ are trainable weights, $u_t$ is a $d$ -dimension vector. The function relu is the rectified linear units BIBREF43 and $\odot $ is element-wise multiply over two vectors. The same Highway Layer is applied to $q_t$ and produces $v_t$ .
Next, $u_t$ and $v_t$ are fed into a Bi-Directional Long Short-Term Memory Network (BiLSTM) BIBREF44 respectively in order to model the temporal interactions between sequence words:
Here we obtain $\mathbf {U} = [u_1^{^{\prime }}, ... , u_n^{^{\prime }}] \in \mathbb {R}^{2d \times n}$ and $\mathbf {V} = [v_1^{^{\prime }}, ... , v_m^{^{\prime }}] \in \mathbb {R}^{2d \times m}$ . Then we feed $\mathbf {U}$ and $\mathbf {V}$ into the attention flow layer BIBREF27 to model the interactions between the input text and query. We obtain the $8d$ -dimension query-aware context embedding vectors $h_1, ... , h_n$ as the result.
After modeling interactions between the input text and queries, we need to enhance the interactions within the input text words themselves especially for the longer text in IE settings. Therefore, we introduce Self-Matching Layer BIBREF29 in our model as
$$\begin{split} o_t &= {\rm BiLSTM}(o_{t-1}, [h_t, c_t]) \\ s_j^t &= w^T {\rm tanh}(W_hh_j+\tilde{W_h}h_t)\\ \alpha _i^t &= {\rm exp}(s_i^t)/\Sigma _{j=1}^n{\rm exp}(s_j^t)\\ c_t &= \Sigma _{i=1}^n\alpha _i^th_i ~. \end{split}$$ (Eq. 20)
Here $W_h, \tilde{W_h} \in \mathbb {R}^{d \times 8d}$ and $w \in \mathbb {R}^d$ are trainable weights, $[h, c]$ is vector concatenation across row. Besides, $\alpha _i^t$ is the attention weight from the $t^{th}$ word to the $i^{th}$ word and $c_t$ is the enhanced contextual embeddings over the $t^{th}$ word in the input text. We obtain the $2d$ -dimension query-aware and self-enhanced embeddings of input text after this step. Finally we feed the embeddings $\mathbf {O} = [o_1, ... , o_n]$ into a Pointer Network BIBREF39 to decode the answer sequence as
$$\begin{split} p_t &= {\rm LSTM}(p_{t-1}, c_t) \\ s_j^t &= w^T {\rm tanh}(W_oo_j+W_pp_{t-1})\\ \beta _i^t &= {\rm exp}(s_i^t)/\Sigma _{j=1}^n{\rm exp}(s_j^t)\\ c_t &= \Sigma _{i=1}^n\beta _i^to_i~. \end{split}$$ (Eq. 21)
Here $\beta _{n+1}^t$ denotes the probability of generating the “ ${\rm eos}$ ” symbol since the decoder also needs to determine when to stop. Therefore, the probability of generating the answer sequence $\textbf {a}$ is as follows
$${\rm P}(\textbf {a}|\mathbf {O}) = \prod _t {\rm P}(a^t | a^1, ... , a^{t-1}, \mathbf {O})~.$$ (Eq. 23)
The input of our model are the words in the input text $x[1], ... , x[n]$ and query $q[1], ... , q[n]$ . We concatenate pre-trained word embeddings from GloVe BIBREF40 and character embeddings trained by CharCNN BIBREF41 to represent input words. The $2d$ -dimension embedding vectors of input text $x_1, ... , x_n$ and query $q_1, ... , q_n$ are then fed into a Highway Layer BIBREF42 to improve the capability of word embeddings and character embeddings as

$$\begin{split} g_t &= {\rm sigmoid}(W_gx_t+b_g) \\ s_t &= {\rm relu } (W_xx_t+b_x) \\ u_t &= g_t \odot s_t + (1 - g_t) \odot x_t~. \end{split}$$ (Eq. 18)
The same Highway Layer is applied to $q_t$ and produces $v_t$ .
Next, $u_t$ and $v_t$ are fed into a Bi-Directional Long Short-Term Memory Network (BiLSTM) BIBREF44 respectively in order to model the temporal interactions between sequence words:
Then we feed $\mathbf {U}$ and $\mathbf {V}$ into the attention flow layer BIBREF27 to model the interactions between the input text and query.
Therefore, we introduce Self-Matching Layer BIBREF29 in our model as

$$\begin{split} o_t &= {\rm BiLSTM}(o_{t-1}, [h_t, c_t]) \\ s_j^t &= w^T {\rm tanh}(W_hh_j+\tilde{W_h}h_t)\\ \alpha _i^t &= {\rm exp}(s_i^t)/\Sigma _{j=1}^n{\rm exp}(s_j^t)\\ c_t &= \Sigma _{i=1}^n\alpha _i^th_i ~. \end{split}$$ (Eq. 20)
Finally we feed the embeddings $\mathbf {O} = [o_1, ... , o_n]$ into a Pointer Network BIBREF39 to decode the answer sequence as

$$\begin{split} p_t &= {\rm LSTM}(p_{t-1}, c_t) \\ s_j^t &= w^T {\rm tanh}(W_oo_j+W_pp_{t-1})\\ \beta _i^t &= {\rm exp}(s_i^t)/\Sigma _{j=1}^n{\rm exp}(s_j^t)\\ c_t &= \Sigma _{i=1}^n\beta _i^to_i~. \end{split}$$ (Eq. 21)
Therefore, the probability of generating the answer sequence $\textbf {a}$ is as follows

$${\rm P}(\textbf {a}|\mathbf {O}) = \prod _t {\rm P}(a^t | a^1, ... , a^{t-1}, \mathbf {O})~.$$ (Eq. 23)
Answer: A pointer network decodes the answer from a bidirectional LSTM with attention flow layer and self-matching layer, whose inputs come from word and character embeddings of the query and input text fed through a highway layer.