# Chinese Neural Question Generation: Augmenting Knowledge into Multiple Neural Encoders
## 3 Our Neural Question Generation Approach
### Passage Compression Step Using the Knowledge Graph
#### 3.2.2 Double Attention-Based Decoder  
We extended previous attention-based question decoder approaches [14; 31] by employing a combination of global and local attention mechanisms to decode the passage, triples and answer information to generate questions. The question words are encoded by using the BERT model since this pre-trained transformer can capture rich contextual embedding of decoded question words. Inspired by [PERSON] and [PERSON]'s work [19] on using the BERT model to decode questions by adding a <MASK> token to capture the previous decoded results, we added a <MASK> token to the end of decoded question words in our model. The question words are denoted as question = ([CLS],[START],Q,[MASK]), where Q is \([q_{cls},q_{start},q_{1}\cdots q_{last},q_{mask}].q_{last}\) is the last decoded question token. The hidden state of the last BERT layer for each corresponding question word is denoted as follows:  
\[s_{\text{question}}=[s_{\text{cls}},s_{\text{start}},s_{1},\dots,s_{\text{last} },s_{mask}] \tag{4}\]  
where \(s_{mask}\) is the hidden state of the current word decoding and \(s_{last}\) refers to the hidden states of the last decoded question token.  
The proposed attention mechanism emphasizes the global attention distribution over encoders and the local attention distribution of each encoder over input characters. The global attention weights indicate how the model spreads out the amount it cares about to different encoders. At time step t, the global attention weights for an \(i\)-th encoder \(a_{global}^{t,i}\) and context vector \(c_{t}\) are calculated as follow:  
\[e_{t,i}=v^{T}\text{tanh}\big{(}W_{h}Encoder_{i}\big{(}a_{local,h}\big{)}+W_{s}s_{\text{ last}}+b_{attn}\big{)}i=1,2,3 \tag{5}\]  
\[a_{global}^{t,i}=softmax\big{(}e^{t,i}\big{)} \tag{6}\]  
\[c_{t}=\sum_{i=1}^{n}a_{global}^{t,i}Encoder_{i}\big{(}a_{local,h}\big{)} \tag{7}\]  
where \(W_{h},W_{s}+b_{attn}\) are all trainable parameters and \(S_{last}\) is the decoded state of the previous question token. \(Encoder_{i}\big{(}a_{local,h}\big{)}\) is the local context for the \(i\)-th encoder which is calculated:  
\[Encoder_{i}=\sum_{c=1}^{n}a_{local}^{t,c}h_{c} \tag{8}\]  
where \(n\) is the number of input tokens in the \(i\)-th encoder and \(a_{local}^{t,c}\) represent the local attention weights for each token \(c\). \(h_{c}\) is the token \(c\)'s hidden state shown in the encoder section. alocal is computed based on \(h_{c}\) and \(S_{l}\) through the concatenate attention mechanism [11]. Based on the previous decoded question token embedding \(w_{last}\), the current decoder state \(s_{mask}\) and the current context vector \(c_{t}\), a readout state \(f_{t}\) is computed by an MLP maxout layer with dropouts. Finally, the readout state is passed to a linear layer and a softmax layer to predict the probabilities of the next word over the decoding vocabulary.  
\[f_{t}=w_{a}w_{last}+w_{b}c_{t}+w_{a}s_{mask} \tag{9}\]\[m_{t}=\left[max\{f_{t,2j-1}f_{t,2j}\}\right]_{j=1,\ldots,d}^{T} \tag{10}\]  
\[p_{vocab}=softmax(w_{0}m_{t}) \tag{11}\]  
During the training phase, the objective is to minimize the difference between the generated question tokens and the reference in the training set. The difference or loss is quantified by using the following negative log likelihood for each question:  
\[\mathcal{L}(\theta)=-\sum_{t=1}^{|Q|+1}log\,P(q_{t}|C,A,K,\{q_{i}\}_{i=0}^{t-1 },\theta) \tag{12}\]  
where \(C\) is the compressed passage tokens, \(A\) is the answer tokens, \(K\) is the knowledge triples, \(\{q_{i}\}\) is the number of decoded question tokens.