# Chinese Neural Question Generation: Augmenting Knowledge into Multiple Neural Encoders
## 4 Experiments
### Machine Evaluation and Results
#### 4.3.1 Comparison Results with Different Question Generation Models  
We compare our model with approaches that have open source codes. Table 4 shows that our proposed Multi-encQG++ model outperformed the BERT-SQR and NQG++MP, which were the state-of-the-art QG models, by at least 7.629 average BLEU score points and by 9.81 ROUGE-L score points in the WebQA dataset. This is mainly due to the passage compression step which has reduced the complexity of the whole QG process. Another reason is the multi-encoder sequence-to-sequence neural model with a double attention mechanism which allows the decoder to focus on the answer and generate correct question words. We will examine the correctness of question words in the Human Evaluation Section.  
The results also showed that BERT-SQR outscored NQG++MP and other question generation models in the dataset, which was consistent with the finding in [PERSON] and [PERSON]'s study [19], indicating the power of BERT contextual input passage representation for question generation. Moreover, mT5 slightly outperformed BERT-SQR, which demonstrated that the encoder-decoder architecture is more suitable than the single BERT-based language model for text generation task [43]. We also note that the NQG++ model outperforms the Trans-Rules and s2s+Att by a large margin in terms of BLEU and ROUGE-L scores. This result confirms the importance of the answer features encoded for question generation proposed by [PERSON] et al. [15]. Interestingly, when the passage compression step is added to the NQG++ model, the resulting NQG++PC model performance decreases because of the loss of contextual information.  
Lastly, we investigated the impact of encoding schemes in Multi-encQG++ and observed that the Multi-encQG++ PT/A model with a separate answer encoder schema outperformed the Multi-encQG++ with one and the other two encoders since the Multi-encQG ++ PT/A pays more attention to the answer when decoding. These results indicated the importance of encoding answer information in a separate encoder.  
It has been observed that the BLEU-4 and BLEU-3 scores were relatively lower than BLEU-1 and BLEU-2 due to the small number of overlapping long text between human question and system question. In fact, the questions generated by neural models tend to use synonyms for the human question where the BLEU evaluation metrics cannot detect it (see an example in Table 5). In addition, human questions have perfect punctuation, such as \"\", ( and ), whereas the system question often does not contain this punctuation. This would cause different tokenization results of some proper terms, such as \"\"(running horse), when computing the BLEU scores. A number of English neural question generation studies reported similar findings [19; 31].  
\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Dataset** & **Model** & **BLEU-1** & **BLEU-2** & **BLEU-3** & **BLEU-4** & **ROUGE-L** \\ \hline \multirow{8}{*}{WebQA} & Trans-Rules & 23.21 & 12.00 & 7.24 & 4.51 & 18.02 \\  & s2s+Att & 23.61 & 12.96 & 7.99 & 5.05 & 19.15 \\  & NQG++ & 28.06 & 15.66 & 9.82 & 6.45 & 23.97 \\  & NQG++PC & 24.65 & 13.04 & 8.09 & 6.08 & 21.16 \\  & NQG++MP & 33.32 & 18.78 & 12.65 & 8.12 & 29.05 \\  & BERT-SQR & 34.14 & 20.13 & 13.45 & 8.98 & 30.47 \\  & mT5 & 36.52 & 21.45 & 15.24 & 9.88 & 32.58 \\  & Single-encQG & 30.12 & 15.38 & 9.35 & 4.92 & 25.72 \\  & Multi-encQG+TA/P & 32.52 & 17.75 & 12.05 & 8.08 & 28.85 \\  & Multi-encQG+PA/T & 31.75 & 16.19 & 11.85 & 7.78 & 27.50 \\  & Multi-encQG+PT/A & 34.10 & 19.76 & 13.21 & 8.37 & 30.13 \\  & Multi-encQG++ & 42.19 & 30.15 & 22.71 & 12.18 & 40.28 \\  & Multi-encQG-Triple & 26.85 & 14.58 & 8.58 & 6.94 & 23.78 \\  & Multi-encQG-answer & 23.98 & 13.21 & 8.34 & 5.28 & 20.65 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Machine evaluation results of different question generation models on the Baidu Q and A dataset. Top 3 ranked triples were incorporated into Multi-encQG models.